| *** achanda has quit IRC | 00:12 | |
| *** Kennan_Vacation2 has quit IRC | 00:25 | |
| *** Kennan_Vacation has joined #kolla | 00:30 | |
| *** Kennan_Vacation is now known as Kennan | 00:34 | |
| *** achanda has joined #kolla | 01:13 | |
| *** dims_ has quit IRC | 01:28 | |
| *** diogogmt has quit IRC | 01:28 | |
| *** diogogmt has joined #kolla | 01:29 | |
| *** Kennan2 has joined #kolla | 01:33 | |
| *** harmw_ has joined #kolla | 01:36 | |
| *** Kennan has quit IRC | 01:38 | |
| *** harmw has quit IRC | 01:38 | |
| *** erkules_ has joined #kolla | 01:39 | |
| *** dimsum__ has joined #kolla | 01:40 | |
| *** erkules has quit IRC | 01:42 | |
| *** dwalsh has joined #kolla | 01:57 | |
| *** CBR09 has joined #kolla | 02:12 | |
| *** CBR09 has quit IRC | 02:16 | |
| *** banix has joined #kolla | 02:27 | |
| *** klint has joined #kolla | 02:42 | |
| *** dimsum__ has quit IRC | 02:48 | |
| *** dimsum__ has joined #kolla | 03:09 | |
| *** dims_ has joined #kolla | 03:10 | |
| *** dwalsh has quit IRC | 03:10 | |
| *** dimsum__ has quit IRC | 03:14 | |
| *** unicell has quit IRC | 03:19 | |
| *** unicell has joined #kolla | 03:19 | |
| *** diogogmt has quit IRC | 03:24 | |
| *** dimsum__ has joined #kolla | 03:25 | |
| *** dimsum__ has quit IRC | 03:27 | |
| *** dims_ has quit IRC | 03:28 | |
| *** achanda has quit IRC | 03:32 | |
| *** achanda has joined #kolla | 03:33 | |
| *** banix has quit IRC | 03:41 | |
| *** achanda has quit IRC | 03:41 | |
| *** alisonh has quit IRC | 03:54 | |
| *** diga has joined #kolla | 03:56 | |
| *** alisonh has joined #kolla | 03:59 | |
| *** dimsum__ has joined #kolla | 04:27 | |
| *** dimsum__ has quit IRC | 04:34 | |
| *** achanda has joined #kolla | 05:12 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fixes inconsistency in ansible inventories https://review.openstack.org/228306 | 05:38 |
|---|---|---|
| *** tfukushima has joined #kolla | 05:42 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Unify commands in documentation https://review.openstack.org/228308 | 05:49 |
| coolsvap | SamYaple, there is no official date for rc's, we can do it today or at max tomorrow i suppose, its upto project how many rc's they need | 06:03 |
| coolsvap | https://wiki.openstack.org/wiki/Liberty_Release_Schedule | 06:03 |
| SamYaple | coolsvap: ah. thanks for the info | 06:03 |
| SamYaple | thats good. i have quite a few patches that need to merge since kolla is b0rked pretty bad right now | 06:03 |
| coolsvap | SamYaple, but the liberty release should be by oct 15 | 06:04 |
| SamYaple | yes | 06:04 |
| *** diga has quit IRC | 06:04 | |
| SamYaple | coolsvap: if you would'nt mind hitting the review queue pretty hard, that would be appreciated | 06:04 |
| coolsvap | SamYaple, yes | 06:05 |
| openstackgerrit | Merged openstack/kolla: Upgrade packages when installing openstack-clients https://review.openstack.org/226167 | 06:20 |
| *** diga has joined #kolla | 06:29 | |
| *** dimsum__ has joined #kolla | 06:31 | |
| Kennan2 | Hi SamYaple: | 06:33 |
| *** Kennan2 is now known as Kennan | 06:33 | |
| SamYaple | hello Kennan2 | 06:33 |
| SamYaple | how are you today? | 06:33 |
| Kennan | I found msg: Destination directory /etc/kolla/nova-libvirt does not exist | 06:33 |
| Kennan | when I tried ansible deploy | 06:33 |
| SamYaple | Kennan: https://review.openstack.org/#/c/228246/ | 06:33 |
| SamYaple | there are actually a few bugs introduced by my sloppy removal of config-external and others trying to rush for the sept 25th deadline. I have put up corrected patches for all issues i have seen | 06:35 |
| *** dimsum__ has quit IRC | 06:36 | |
| Kennan | OK SamYaple: let me try that patch in my env | 06:37 |
| SamYaple | Kennan: due to related changes openvswitch is also broken. | 06:40 |
| SamYaple | i have a patchseries up there to fix all that | 06:41 |
| Kennan | SamYaple: do you mean this /opt/kolla/config-external.sh: No such file or directory ? | 06:43 |
| SamYaple | Kennan: correct | 06:44 |
| SamYaple | some of the services (openvswitch) are trying to source tha tfile and it doesnt exist | 06:45 |
| SamYaple | it never di anything, but the fact that it doesnt exist is bombing it out | 06:45 |
| SamYaple | coolsvap: fyi all of the above problems in master are fix in this commit (and the following patchsets) https://review.openstack.org/#/c/228244/ | 06:45 |
| SamYaple | i would call them top priority | 06:45 |
| coolsvap | SamYaple, yes, i am looking at those | 06:46 |
| SamYaple | I have tested the whole stream, but not neccessarily the individual patches | 06:47 |
| SamYaple | for example, i break ceph between two of those patches, but nothing else breaks | 06:47 |
| coolsvap | SamYaple, alright thanks for the info | 06:48 |
| *** itsuugo has joined #kolla | 06:56 | |
| *** achanda has quit IRC | 06:56 | |
| *** inc0 has joined #kolla | 06:59 | |
| *** achanda has joined #kolla | 06:59 | |
| *** achanda has quit IRC | 07:02 | |
| inc0 | howdy folks | 07:03 |
| SamYaple | inc0! wonderful | 07:03 |
| SamYaple | neutron-server b0rked | 07:03 |
| SamYaple | logging related | 07:03 |
| coolsvap | Kennan, though its not a Kolla patchset, but would you mind opening a tracker bug associated with the check-requirements job change set? | 07:03 |
| inc0 | oh? how so? | 07:03 |
| SamYaple | inc0: ill fetch you the error. its wierd. i think its a neutron-server bug | 07:03 |
| SamYaple | but its broke for sure | 07:03 |
| Kennan | coolsvap: do you mean I open a bug in kolla ? | 07:04 |
| coolsvap | Kennan, yes | 07:05 |
| Kennan | ok I would open a bug, and add comment in bugs about this link(code change) | 07:05 |
| coolsvap | a (tracker) bug | 07:05 |
| coolsvap | thanks! | 07:05 |
| Kennan | hi coolsvap: | 07:10 |
| Kennan | I did not find bug type during bug creation | 07:10 |
| Kennan | https://bugs.launchpad.net/kolla/+bug/1500329 | 07:10 |
| openstack | Launchpad bug 1500329 in kolla "[Tracker]Kolla needs related check requirements job and add it in requirements family" [Undecided,New] | 07:10 |
| Kennan | I just create this bug as usual bug | 07:11 |
| coolsvap | Kennan, its not a bug type as such | 07:11 |
| coolsvap | its tracker in the sense of its purpose, the changes you are making are not lost and are tracked | 07:11 |
| Kennan | so is it ok for this bug I created ? | 07:12 |
| coolsvap | yes | 07:12 |
| Kennan | ok. thx | 07:13 |
| *** tfukushima has quit IRC | 07:14 | |
| *** tfukushima has joined #kolla | 07:14 | |
| *** itsuugo has quit IRC | 07:16 | |
| *** itsuugo has joined #kolla | 07:17 | |
| *** mbound has joined #kolla | 07:20 | |
| *** akwasnie has joined #kolla | 07:30 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Swift logging to rsyslog https://review.openstack.org/227881 | 07:36 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 07:36 |
| inc0 | SamYaple, do we want logfile per every swift service? | 07:37 |
| SamYaple | yes, like everything else | 07:37 |
| inc0 | duh, I don't like swift already | 07:38 |
| SamYaple | yup | 07:38 |
| SamYaple | inc0: btw, quite a few bug fixes/patches in the queue to fix the broken-in-several-ways master that is kolla right now | 07:38 |
| *** CBR09 has joined #kolla | 07:39 | |
| mlil | Hi again! Since last time I asked for help I have now managed to get further, but now I've stopped on another error message that I'm not sure what to do about: | 07:39 |
| mlil | failed: [controller1.opnfv] => {"failed": true, "parsed": false} | 07:39 |
| mlil | Traceback (most recent call last): | 07:39 |
| mlil | File "/home/opnfv/.ansible/tmp/ansible-tmp-1443198399.34-42584921028986/docker", line 3185, in <module> | 07:39 |
| mlil | main() | 07:39 |
| mlil | File "/home/opnfv/.ansible/tmp/ansible-tmp-1443198399.34-42584921028986/docker", line 1482, in main | 07:39 |
| mlil | docker_api_version = dict(required=False, default=DEFAULT_DOCKER_API_VERSION, type='str'), | 07:39 |
| mlil | NameError: global name 'DEFAULT_DOCKER_API_VERSION' is not defined | 07:39 |
| mlil | I guess I have missed some configuration, the question is which configuration? | 07:40 |
| inc0 | SamYaple, which ones are most pressing | 07:41 |
| inc0 | ? | 07:41 |
| CBR09 | Did you install docker-py>=1.2.0? | 07:44 |
| *** jmccarthy has quit IRC | 07:46 | |
| CBR09 | mlil: you can try install docker-py and run again | 07:46 |
| *** jmccarthy has joined #kolla | 07:47 | |
| mlil | Hmm, Ubuntu doesn't know what "docker-py", but I know I installed "docker-engine". I'll do some googling to see if the packet just needs another name... | 07:47 |
| CBR09 | mlil: try pip install docker-py | 07:50 |
| mlil | Well, I did find python-docker package, I'll see if that works. If not, I'll try pip. | 07:51 |
| SamYaple | mlil: what version of ansible are you rinnung? | 08:05 |
| SamYaple | mlil: any packaged versoin of docker-py will likely be too old | 08:05 |
| SamYaple | inc0: https://review.openstack.org/#/c/228244/ starting with that patchstream and up are most pressing | 08:06 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Swift logging to rsyslog https://review.openstack.org/227881 | 08:06 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 08:06 |
| mlil | Well, the python-docker itself is of version 0.5.3-2. So perhaps you're right SamYaple, it's too old. | 08:07 |
| inc0 | SamYaple, care to review plz? Murano configs are in swift patchset, but I won't split it now I guess.. | 08:07 |
| mlil | SamYaple: ansible is of version 1.9.3 | 08:07 |
| SamYaple | mlil: that is a good version, my guess is just too old docker-py | 08:08 |
| SamYaple | like CBR09 said | 08:08 |
| SamYaple | inc gotta dock you for the murano in that patchset | 08:08 |
| SamYaple | inc0: ^ | 08:08 |
| SamYaple | you end up double adding them | 08:08 |
| inc0 | no, look at murano | 08:09 |
| inc0 | changes in config is not there any more | 08:09 |
| inc0 | that's how git works | 08:09 |
| inc0 | so when these 2 merges, it will be exactly the same effect | 08:09 |
| SamYaple | you are wrong | 08:09 |
| SamYaple | https://review.openstack.org/#/c/227900/3/ansible/roles/common/templates/rsyslog.conf.j2 | 08:10 |
| inc0 | hmm, strange then locally it looks just fine | 08:10 |
| inc0 | well whatever, let me get this right | 08:10 |
| SamYaple | inc0: other comment in swift patch | 08:10 |
| SamYaple | please review | 08:10 |
| SamYaple | you didnt respond to the query before | 08:10 |
| *** CBR09 has quit IRC | 08:11 | |
| inc0 | ahh must've missed that | 08:11 |
| SamYaple | so about log_name and log_level | 08:11 |
| *** athomas has joined #kolla | 08:11 | |
| *** gfidente has joined #kolla | 08:12 | |
| SamYaple | all the other patches have just use_syslog and syslog_facility | 08:12 |
| inc0 | log name is required | 08:12 |
| SamYaple | thats going to be a problem | 08:12 |
| inc0 | yeah, that's swift, that's different | 08:12 |
| SamYaple | are you sure they dont import oslo.logging | 08:12 |
| inc0 | swift doesn't follow convention | 08:12 |
| inc0 | pretty sure | 08:12 |
| SamYaple | hmmm then youll have to do a bit more templating in the swift conf | 08:14 |
| SamYaple | to setup the file name per service | 08:14 |
| inc0 | it's not that bad | 08:15 |
| SamYaple | actually inc0 you should be able to use a variable for that | 08:17 |
| inc0 | that's what I'm doing | 08:17 |
| inc0 | service name | 08:17 |
| inc0 | just let me test it out | 08:17 |
| inc0 | (forgot to include that to patchset) | 08:17 |
| SamYaple | log_facility = {{ service_name }} | 08:18 |
| SamYaple | yea thatll work | 08:18 |
| inc0 | log_name, but yeah | 08:18 |
| inc0 | that was my idea | 08:18 |
| inc0 | hmm...haproxy is busted | 08:19 |
| SamYaple | lots of things are | 08:19 |
| SamYaple | see review stream above | 08:19 |
| inc0 | damn. | 08:20 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Swift logging to rsyslog https://review.openstack.org/227881 | 08:33 |
| *** dimsum__ has joined #kolla | 08:33 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 08:33 |
| inc0 | guys, I'll leave log_level for now as I can't really test without it currently | 08:34 |
| *** erkules_ is now known as erkules | 08:37 | |
| *** erkules has quit IRC | 08:37 | |
| *** erkules has joined #kolla | 08:37 | |
| *** dimsum__ has quit IRC | 08:38 | |
| SamYaple | but you tested murano right inc0? | 08:38 |
| SamYaple | ;) | 08:38 |
| inc0 | 2 days ago, yes | 08:39 |
| SamYaple | lol look at it again | 08:39 |
| inc0 | damn, forgot to ctrl+s | 08:39 |
| SamYaple | :D | 08:40 |
| inc0 | shame on me | 08:40 |
| SamYaple | shame shame shame | 08:40 |
| SamYaple | pbourke_: you around bud? | 08:40 |
| pbourke_ | SamYaple: yeah, hi | 08:40 |
| pbourke_ | hows it going | 08:40 |
| SamYaple | hey man. we gots some major breaking bugs in master right now | 08:41 |
| pbourke_ | :/ | 08:41 |
| SamYaple | ill take the blame, but this patchset fixes it all | 08:41 |
| SamYaple | https://review.openstack.org/#/c/228244/ | 08:41 |
| SamYaple | it corrects 4-5 bugs and then there are more corrections that arent as urgent in the queue | 08:41 |
| SamYaple | if you get some time we could use reviews across all of kolla today | 08:42 |
| pbourke_ | sure thing | 08:42 |
| SamYaple | thanks man | 08:42 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 08:43 |
| SamYaple | come on inc0! | 08:43 |
| inc0 | damn.. | 08:44 |
| SamYaple | haha one of those days | 08:44 |
| inc0 | it's called Monday | 08:45 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 08:45 |
| SamYaple | note to all: https://review.openstack.org/#/c/227784/ that is a patch for project-config to fix DocImpact for Kolla, external +1s for pressure are welcome! | 08:46 |
| pbourke_ | SamYaple: https://review.openstack.org/#/c/228244/1/ansible/roles/neutron/templates/openvswitch-db-server.json.j2 - should this be using rsyslog? | 08:51 |
| SamYaple | pbourke_: yea but thats not what that patch is for | 08:51 |
| SamYaple | its just a copy from the config-external code | 08:51 |
| SamYaple | rather start.sh | 08:51 |
| SamYaple | hey wait a minute arent you the one that complains when i combine unrelated patches? | 08:52 |
| inc0 | no, that was me | 08:52 |
| SamYaple | i think pbourke_ has too... | 08:53 |
| pbourke_ | i do :) | 08:53 |
| SamYaple | either way, there are still a few services that need to be rsyslog'd pbourke_ | 08:53 |
| SamYaple | openvswitch and haproxy among them | 08:53 |
| pbourke_ | makes sense if it was existing, that's fine | 08:53 |
| inc0 | yeah, one thing at the time plz | 08:53 |
| openstackgerrit | Merged openstack/kolla: Replace config-external - openvswitch https://review.openstack.org/228244 | 08:55 |
| *** exploreshaifali has joined #kolla | 08:59 | |
| SamYaple | hey pbourke_ when you file/close bugs would you mind setting the milestone and assigning it to yourself? it helps tracking and statistics | 09:00 |
| SamYaple | otherwise me and sdake are going back through and fixing it | 09:00 |
| pbourke_ | ok | 09:01 |
| *** inc0 has quit IRC | 09:03 | |
| SamYaple | 74 bugs fixed for rc1 release :) | 09:04 |
| SamYaple | probably more if we merge in queue stuff | 09:04 |
| *** inc0 has joined #kolla | 09:05 | |
| openstackgerrit | Merged openstack/kolla: Swift logging to rsyslog https://review.openstack.org/227881 | 09:08 |
| *** achanda has joined #kolla | 09:10 | |
| pbourke_ | SamYaple: for removing set_configs - it seems not every component is using the new config mechanism yet. so is it correct to remove set_configs for these? | 09:10 |
| pbourke_ | SamYaple: e.g. ceph https://review.openstack.org/#/c/228247/1/docker/ceph/ceph-osd/start.sh | 09:11 |
| SamYaple | pbourke_: yes but two patches up i fix that | 09:11 |
| SamYaple | thats the only one | 09:11 |
| pbourke_ | ah I keep forgetting to check the dependencies | 09:11 |
| SamYaple | no it depends on that. i did the ordering wrong | 09:11 |
| SamYaple | basically everythings is broken right now, only ceph will be broken once the set_configs removal merges | 09:12 |
| SamYaple | then the following patches fix ceph | 09:12 |
| *** inc0 has quit IRC | 09:14 | |
| *** inc0_ has joined #kolla | 09:14 | |
| openstackgerrit | Merged openstack/kolla: Replace config-external - rsyslog https://review.openstack.org/228245 | 09:14 |
| mlil | SamYaple: I now have non-working pip, regardless if I reinstall it or not, can't even get version text from it... Seems like I have to re-install my nodes. Just 5 machines to go so I should be finished any day now... I guess this was caused by changing my mind over how to install stuff. We'll see how the re-install goes. But thanks for the help so far! | 09:15 |
| *** sdake has joined #kolla | 09:15 | |
| SamYaple | mlil: if this is ubuntu you will need to relog from your user | 09:15 |
| SamYaple | when pip upgrading it will change pip install location and bash hashing wont keep up | 09:16 |
| SamYaple | easiest fix is to relog | 09:16 |
| SamYaple | pbourke_: can you update/close this bug if it is not valid? https://bugs.launchpad.net/kolla/+bug/1498456 | 09:18 |
| openstack | Launchpad bug 1498456 in kolla "The default vagrant provider should be virtualbox" [Undecided,New] | 09:18 |
| mlil | SamYaple: Oh, so I don't need to re-install then? Yepp, just needed to log out and in again. Interesting... Thanks for saving me the effort of re-installation! ;-) | 09:19 |
| SamYaple | yea the dist installed pip is in /usr/bin/pip, but when you `pip install -U pip` it gets installed in /usr/local/bin/pip | 09:19 |
| SamYaple | bash hashes things in such a way that it won't pick up on the new pip | 09:20 |
| SamYaple | couple of ways to fix it, but relogging is easiest | 09:20 |
| SamYaple | vagrant people, someone please grab this bug: https://bugs.launchpad.net/kolla/+bug/1496920 | 09:22 |
| openstack | Launchpad bug 1496920 in kolla "vagrant VM can not create virbr0" [Undecided,New] | 09:22 |
| SamYaple | Kennan: are yo uaround? | 09:25 |
| Kennan | Hi SamYaple: | 09:25 |
| SamYaple | hi! docs question | 09:25 |
| SamYaple | https://review.openstack.org/#/c/228308/ | 09:25 |
| SamYaple | coolsvap has asked me to add a NOTE/warning | 09:25 |
| SamYaple | is there special blocks for that? | 09:25 |
| SamYaple | like "NOTE: deployer is responsible to running with the appropriate privleges, some commands require root elevation" | 09:26 |
| Kennan | yes, add something like that, some commands require run with sudo | 09:27 |
| SamYaple | Kennan: right but is there a special command block to get it to appear as a special warning/note section at docs.openstack.org | 09:29 |
| * SamYaple doesn't know docs | 09:29 | |
| pbourke_ | tbh if you feel we have to add a note like that, the commands should stay as is | 09:30 |
| Kennan | SamYaple: not know any special commands. | 09:30 |
| pbourke_ | but its not a big deal | 09:31 |
| SamYaple | pbourke_: i dont think we need it | 09:32 |
| SamYaple | but i dont care really | 09:32 |
| pbourke_ | same | 09:32 |
| inc0_ | yay all of openstack services are merged | 09:34 |
| inc0_ | rsyslog loggin I mean | 09:35 |
| openstackgerrit | Merged openstack/kolla: Create nova-libvirt directory before copying files https://review.openstack.org/228246 | 09:35 |
| *** dimsum__ has joined #kolla | 09:36 | |
| SamYaple | thanks Kennan! | 09:37 |
| openstackgerrit | Merged openstack/kolla: Fixes inconsistency in ansible inventories https://review.openstack.org/228306 | 09:42 |
| openstackgerrit | Merged openstack/kolla: Remove all fail sections for Ubuntu https://review.openstack.org/218225 | 09:42 |
| *** dimsum__ has quit IRC | 09:45 | |
| *** dimsum__ has joined #kolla | 09:51 | |
| openstackgerrit | Merged openstack/kolla: Allow different defaults for different providers https://review.openstack.org/227147 | 09:51 |
| inc0_ | guys, can you build ubuntu source openstack-base? | 09:51 |
| SamYaple | inc0_: yea why | 09:52 |
| inc0_ | Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main binutils amd64 2.24-5ubuntu13 | 09:52 |
| inc0_ | seems like packages are busted somehow | 09:52 |
| SamYaple | did you build with --no-cache | 09:52 |
| openstackgerrit | Zhao Lei proposed openstack/kolla: Remove quotes from subshell call in bash script https://review.openstack.org/226734 | 09:52 |
| inc0_ | trying | 09:53 |
| SamYaple | im going a full rebuild for you too inc0_ | 09:54 |
| inc0_ | it never hurts | 09:55 |
| *** achanda has quit IRC | 09:55 | |
| sdake | morning | 09:55 |
| inc0_ | hi sdake, feeling better? | 09:56 |
| sdake | still alive | 09:56 |
| SamYaple | morning sdake | 09:56 |
| SamYaple | were working through them bugs and reviews | 09:56 |
| SamYaple | quite the team | 09:56 |
| SamYaple | inc0_: openstack-base builds for me no problem | 09:59 |
| SamYaple | my guess is out-of-date apt cache pulling old package | 09:59 |
| inc0_ | probably | 10:00 |
| inc0_ | still building for me | 10:00 |
| inc0_ | but we have network based on ToP - TCP over pidgeons | 10:02 |
| SamYaple | ah good ole reliable RFC 1149 | 10:04 |
| inc0_ | exactly | 10:05 |
| *** dwalsh has joined #kolla | 10:07 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Implement a install_type and install_metatype https://review.openstack.org/224841 | 10:07 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Significant refactor of base image https://review.openstack.org/222893 | 10:07 |
| openstackgerrit | Merged openstack/kolla: Bind novnc to single interface https://review.openstack.org/228238 | 10:10 |
| openstackgerrit | Steven Dake proposed openstack/kolla: eat doesn't need to specify python-oslo-reports https://review.openstack.org/227967 | 10:11 |
| *** exploreshaifali has quit IRC | 10:11 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Heat doesn't need to specify python-oslo-reports https://review.openstack.org/227967 | 10:12 |
| * sdake wtb functional gating | 10:13 | |
| inc0_ | would be nice, yeah | 10:14 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Implement a install_type and install_metatype https://review.openstack.org/224841 | 10:17 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Heat doesn't need to specify python-oslo-reports https://review.openstack.org/227967 | 10:17 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Significant refactor of base image https://review.openstack.org/222893 | 10:17 |
| sdake | what is going on with the centos source gate? | 10:20 |
| sdake | SamYaple didy ou fix the setuptools thing mandre was working on? | 10:22 |
| SamYaple | sdake: mandre did. i believe it merged | 10:22 |
| inc0_ | sdake, it's undeterministic | 10:22 |
| inc0_ | it breaks from time to time | 10:22 |
| sdake | inc0 no it was breaking all the time | 10:22 |
| openstackgerrit | Merged openstack/kolla: Unify commands in documentation https://review.openstack.org/228308 | 10:22 |
| inc0_ | https://review.openstack.org/#/c/227900/ this is 2hrs ago and it works | 10:23 |
| SamYaple | sdake there is _something_ in the centos source base that is installing python things. find it and kill it | 10:23 |
| SamYaple | also, someone needs to move the base files for centos to openstack-base, or I am going to and i can't gaurantee non-breakage | 10:23 |
| inc0_ | I don't like non-breakage just before release... | 10:24 |
| sdake | what is pending work for rc1 while we have this discusison | 10:24 |
| SamYaple | that is pending work | 10:25 |
| SamYaple | https://blueprints.launchpad.net/kolla/+spec/openstack-common-container | 10:25 |
| sdake | i get that, what else is pending work | 10:26 |
| inc0_ | logging container if we include non-openstack services | 10:26 |
| inc0_ | all of openstack is done and merged | 10:26 |
| openstackgerrit | Merged openstack/kolla: Remove quotes from subshell call in bash script https://review.openstack.org/226734 | 10:26 |
| sdake | i've been out of action for about 3 weeks with back to back flu or reallybad colds | 10:27 |
| sdake | so if someone could bring me up to date on what is left for rc1 it would be helpful :) | 10:27 |
| sdake | sam sent me an email yesterday about some breakage in teh code base , is all that fixed up? | 10:27 |
| SamYaple | sdake: some patches left to merge | 10:28 |
| SamYaple | https://review.openstack.org/#/c/228247/ | 10:28 |
| SamYaple | that one is the worst offender | 10:28 |
| sdake | i'll review that now | 10:28 |
| sdake | anything not in the queue that needs attention? | 10:28 |
| sdake | besides the common centos container? | 10:28 |
| inc0_ | ironic is almost merged, but ansible part is still in review | 10:30 |
| inc0_ | ceph was merged | 10:30 |
| SamYaple | yea that ansible stuff needs to merge so i can fix config-external on it | 10:30 |
| inc0_ | yup, I've noticed it's still old-style | 10:30 |
| sdake | samyaple this is -1'ed https://review.openstack.org/#/c/228248/ | 10:32 |
| *** pbourke_ has quit IRC | 10:33 | |
| SamYaple | sdake: you did? | 10:33 |
| SamYaple | pbourke did | 10:33 |
| sdake | no i did not | 10:33 |
| SamYaple | sorry i injected letters in my brain :) | 10:33 |
| *** pbourke_ has joined #kolla | 10:34 | |
| openstackgerrit | Merged openstack/kolla: remove set_configs https://review.openstack.org/228247 | 10:34 |
| *** diga has quit IRC | 10:35 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Allow conf files to be optional https://review.openstack.org/228248 | 10:36 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert ceph to json https://review.openstack.org/228249 | 10:36 |
| sdake | pbourke_ can you review https://review.openstack.org/#/c/224841/ and its children please | 10:38 |
| sdake | samyaple i'll take care o the common refactor for centos | 10:38 |
| openstackgerrit | Merged openstack/kolla: Stop haproxy from listening on ports it shouldnt https://review.openstack.org/228233 | 10:38 |
| SamYaple | thank you sdake | 10:38 |
| SamYaple | let me know if you have questions, but you should be able to clean up the base alot like that | 10:39 |
| sdake | inc0_ can yo utell me more of what is needed for logging container for non-openstack services? | 10:39 |
| SamYaple | basically just repo enablement at that point | 10:39 |
| sdake | right | 10:39 |
| sdake | that is what i want the base image to be | 10:39 |
| SamYaple | inc0_: sdake: i have a patch for openvswitch and haproxy for syslog | 10:39 |
| SamYaple | im testing | 10:39 |
| openstackgerrit | Merged openstack/kolla: bind haproxy to single interface https://review.openstack.org/228237 | 10:40 |
| inc0_ | ovs, mariadb, rabbitmq, haproxy | 10:40 |
| inc0_ | libvirt | 10:40 |
| sdake | sounds like 2 of 4 are being sorted out | 10:40 |
| sdake | 2 of 5 | 10:40 |
| SamYaple | also, the logging is... wierd. it doesnt currently log tracebacks into the log inc0_ | 10:40 |
| SamYaple | it only logs those to stderr like always | 10:41 |
| inc0_ | SamYaple, not these which stops service from working | 10:41 |
| SamYaple | (so docker logs neutroN_server) | 10:41 |
| inc0_ | if you can't even run service, it won't use rsyslog | 10:41 |
| sdake | does deployment work atm? | 10:41 |
| SamYaple | sdake: neutron_server is borked due to logging | 10:41 |
| SamYaple | i think its a neutron-server bug though | 10:41 |
| SamYaple | inc0_: no all services dont log tracebacks to log | 10:42 |
| sdake | so can we get it working | 10:42 |
| SamYaple | its not configured correctly | 10:42 |
| sdake | neutron server taht is | 10:42 |
| inc0_ | ok I'll look into it, just tell me reproduction steps plz | 10:42 |
| SamYaple | yea i think its a neutron bug though. i dug into it a little bit | 10:42 |
| sdake | if we don't have deployment working i can't test a refactor of base | 10:42 |
| inc0_ | or even better - file a but;) | 10:42 |
| SamYaple | inc0_: create an exception, any kind | 10:42 |
| sdake | file a but? | 10:42 |
| SamYaple | thats how he likes it dont judge | 10:42 |
| inc0_ | I've seen traces in log | 10:42 |
| sdake | ;=) | 10:43 |
| inc0_ | but I'll confirm that | 10:43 |
| SamYaple | inc0_: ok ill double check too | 10:43 |
| sdake | SamYaple can you fix this neutron server thing so i can get deployment working with a bsae refactor? | 10:44 |
| *** tfukushima has quit IRC | 10:44 | |
| SamYaple | sdake: I couldn't fix it, no. i worked on it for about 30 minutes before other things | 10:45 |
| SamYaple | you can augment the file ot turn of logging | 10:45 |
| sdake | what happens exactly? | 10:46 |
| sdake | neutron sserver crashes or something? | 10:46 |
| SamYaple | run it youll see the stack trace, but like i said it appears to be a bug | 10:46 |
| sdake | rc1 has been snapped of neutron | 10:48 |
| sdake | i wonder if they are doing an rc2 | 10:48 |
| openstackgerrit | Merged openstack/kolla: Implement a install_type and install_metatype https://review.openstack.org/224841 | 10:49 |
| sdake | pbourke_ can you ack https://review.openstack.org/#/c/227967/ please | 10:50 |
| sdake | samyaple on https://review.openstack.org/#/c/228248/2/docker/base/set_configs.py | 10:53 |
| SamYaple | yea | 10:53 |
| sdake | if validate_source returns false, nothign happens in the set_configs code? | 10:53 |
| SamYaple | it returns to L162 and continues in that for loop to the next file | 10:54 |
| SamYaple | basically it just skips the copy/set perms step | 10:54 |
| openstackgerrit | Merged openstack/kolla: Significant refactor of base image https://review.openstack.org/222893 | 10:54 |
| sdake | wouldn't we prefer exiting over continuing on in a failed situation? | 10:54 |
| sdake | what is the "optional" thing for exactly? | 10:54 |
| SamYaple | sdake: look at the next patch | 10:54 |
| SamYaple | or the commit msg for that patch | 10:54 |
| sdake | ok will do | 10:54 |
| sdake | SamYaple does ceph not have configurable ports? | 10:57 |
| SamYaple | yes and no, its not as simple as that to the point im not ok with including them just yet | 10:58 |
| sdake | ok so outstanding work is | 10:59 |
| sdake | 1) neutron backtrace | 10:59 |
| sdake | 2) syslog is missing 5 services for logging | 10:59 |
| sdake | 3) openstack base needs a refactor for centos | 10:59 |
| SamYaple | a few non-ansible openstack services still have config-external | 11:00 |
| SamYaple | but thats not breaking, just work needed to remove them | 11:00 |
| SamYaple | (work ive started) | 11:00 |
| SamYaple | the only thing breaking is 1 | 11:00 |
| sdake | well i'm not branching on a doa histroy ;) | 11:00 |
| SamYaple | worst case we can turn off logging for neutron-server and just put it on the todo list | 11:01 |
| sdake | sounds gtm | 11:01 |
| SamYaple | would like to attempt a fix though | 11:01 |
| *** inc0_ has quit IRC | 11:04 | |
| *** inc0 has joined #kolla | 11:05 | |
| sdake | i think the plan then is to branch around 10am PST on tuesday | 11:06 |
| sdake | (in 24 hours) | 11:07 |
| *** inc0 has quit IRC | 11:07 | |
| sdake | once we have done a fix for neutron-server | 11:07 |
| sdake | we cn backport other changes that don't make it in time | 11:07 |
| *** inc0 has joined #kolla | 11:07 | |
| sdake | as part of rc2 | 11:07 |
| SamYaple | cool | 11:07 |
| openstackgerrit | Merged openstack/kolla: Heat doesn't need to specify python-oslo-reports https://review.openstack.org/227967 | 11:07 |
| sdake | no luke, I am your Father | 11:10 |
| sdake | serach your feelings you know it to be true! | 11:10 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Rename ovs to openvswitch https://review.openstack.org/228404 | 11:13 |
| SamYaple | im really hoping to get that merged because if we dont it will cause lots of confusion later to change and i dont want confusion but i dont want it to stay like it is | 11:14 |
| SamYaple | it breaks my regex all the time (doesnt' build openvswitch) | 11:14 |
| *** inc0 has quit IRC | 11:14 | |
| * SamYaple cross his fingers on the last minute change.... | 11:14 | |
| *** inc0_ has joined #kolla | 11:14 | |
| *** tfukushima has joined #kolla | 11:18 | |
| sdake | does anyone want to take this over | 11:20 |
| sdake | https://review.openstack.org/#/c/216112/ | 11:20 |
| sdake | i think tim decided he wasn't going to fix it :) | 11:21 |
| SamYaple | sdake: I could never reproduce that | 11:22 |
| openstackgerrit | Merged openstack/kolla: Allow conf files to be optional https://review.openstack.org/228248 | 11:22 |
| sdake | samyaple one way is to killall -9 httpd | 11:22 |
| sdake | via docker exec | 11:23 |
| SamYaple | will attemp | 11:24 |
| SamYaple | t | 11:24 |
| sdake | i dont quite get why we are using pid files anyway | 11:24 |
| sdake | pid file is to protect against a process running twice | 11:24 |
| sdake | our ssytem could never let that situation occur | 11:25 |
| SamYaple | yea for httpd/apache we should remove that in the config | 11:25 |
| SamYaple | does keystone even bind out run? | 11:25 |
| openstackgerrit | Merged openstack/kolla: Improve cleanup script https://review.openstack.org/217706 | 11:25 |
| sdake | i am not sure SamYaple | 11:25 |
| openstackgerrit | Merged openstack/kolla: Convert ceph to json https://review.openstack.org/228249 | 11:26 |
| SamYaple | sdake: my 10gb and 750 ssds are coming in tomorrow | 11:26 |
| sdake | nice | 11:26 |
| sdake | the 750 ssd rocks | 11:26 |
| SamYaple | yea i got one for each of my servers | 11:26 |
| sdake | i ordered a 8 core 2670 2nd cpu for my system | 11:27 |
| SamYaple | going ot split it between ceph cache tier and docker builds | 11:27 |
| sdake | i have ram to core inbalance | 11:27 |
| SamYaple | yea i run dual Intel(R) Xeon(R) CPU E5-2687W v2 | 11:27 |
| sdake | that the 16 hyperthread cores will fix | 11:27 |
| sdake | is that an 8 core? | 11:27 |
| SamYaple | yup | 11:27 |
| *** inc0_ has quit IRC | 11:28 | |
| SamYaple | 4ghz | 11:28 |
| sdake | nice | 11:28 |
| sdake | those are super epxensive as I recall | 11:28 |
| SamYaple | its lightning | 11:28 |
| sdake | I went for cheap 2.6ghz | 11:28 |
| SamYaple | i do all my video encoding on it | 11:28 |
| SamYaple | yea its worth it. i crush me some numbers for sure | 11:28 |
| sdake | xeon smokes i7 anyway regardless of ghz | 11:28 |
| SamYaple | i think it has all those nice extra instruction sets for things | 11:28 |
| SamYaple | better silicon too | 11:29 |
| SamYaple | not as much variance | 11:29 |
| sdake | the main thing is xeon has a 20 or 25mb cache | 11:29 |
| *** akwasnie has quit IRC | 11:29 | |
| sdake | that is a huge differentiating factor | 11:29 |
| SamYaple | just L3 though | 11:29 |
| SamYaple | that breaks cross processor anyway | 11:29 |
| sdake | still huge ;) | 11:29 |
| SamYaple | so you have to do your affinity correctly | 11:29 |
| sdake | numa should sort that out iirc | 11:30 |
| SamYaple | openstack doesnt default to respecting numa IIRC | 11:30 |
| sdake | if there are onfig options that i probably something we will want to reoslve | 11:30 |
| SamYaple | we would need to verify it first, but yea safe defaults are cool | 11:31 |
| sdake | inc0 rebase https://review.openstack.org/#/c/227900/ plz | 11:32 |
| SamYaple | oh boy. look at that review queue | 11:32 |
| SamYaple | https://review.openstack.org/#/q/status:open+project:openstack/kolla,n,z | 11:32 |
| SamYaple | sdake: i went through most of the bugs and blueprints earlier | 11:35 |
| SamYaple | we can still solve some for rc1 | 11:35 |
| SamYaple | but none are major i believe | 11:35 |
| SamYaple | https://bugs.launchpad.net/kolla/+bug/1485742 | 11:36 |
| openstack | Launchpad bug 1485742 in kolla "global search and replace /opt/kolla with /var/lib/kolla" [Critical,Confirmed] - Assigned to Sam Yaple (s8m) | 11:36 |
| SamYaple | now that set_config is all merged, I can do this bug | 11:36 |
| *** gfidente has quit IRC | 11:36 | |
| SamYaple | you want this before rc1 sdake? | 11:36 |
| sdake | yes | 11:36 |
| sdake | or before liberty | 11:37 |
| SamYaple | k should be quick, then I can test | 11:37 |
| sdake | so that is a #4 item of things to do | 11:37 |
| sdake | probably #2 on priority list | 11:37 |
| sdake | i am pushing master atm | 11:38 |
| sdake | once it wraps up i'll do a deploy | 11:38 |
| SamYaple | psuhing where? | 11:38 |
| sdake | to my registry | 11:38 |
| SamYaple | cool | 11:38 |
| SamYaple | sdake: https://bugs.launchpad.net/kolla/+bug/1485704 | 11:39 |
| openstack | Launchpad bug 1485704 in kolla "remove cinder_volume_driver config option from globals.yml" [Medium,Triaged] - Assigned to Ryan Hallisey (rthall14) | 11:39 |
| SamYaple | need your reply | 11:39 |
| sdake | did the docker cats ever sort out the push operation for v2 registry? | 11:39 |
| SamYaple | in v1.9 i believe there are some improvements | 11:40 |
| SamYaple | but the big improvement got reverted xD | 11:40 |
| SamYaple | inc0 sdake neutron-server stacktrace http://paste.fedoraproject.org/272182/44047714/ | 11:41 |
| SamYaple | seems busted to me | 11:41 |
| SamYaple | sdake: neutron_agents also busted, just masked by supervisord | 11:43 |
| SamYaple | neutron syslog logging may be busted as a whole | 11:44 |
| SamYaple | hmmmm inc0 I am getting no logs from any services now | 11:45 |
| sdake | inc0 is not in irc | 11:45 |
| SamYaple | ahh you remove the privleged: True from the container! for shame! | 11:45 |
| SamYaple | hold on patch incoming | 11:46 |
| *** itsuugo has quit IRC | 11:47 | |
| *** shardy_ has joined #kolla | 12:00 | |
| *** shardy has quit IRC | 12:00 | |
| *** itsuugo has joined #kolla | 12:03 | |
| sdake | SamYaple looks to me that a possible issue is neutron either supports logging to a file/directory OR syslog | 12:04 |
| sdake | but not both at th esame time | 12:04 |
| sdake | and we are likely specifying both with the dist setup | 12:04 |
| SamYaple | well make the change and test, i havent figured it out and i got pulled away | 12:05 |
| sdake | the facility is thus not being setup | 12:05 |
| *** shardy_ has quit IRC | 12:05 | |
| sdake | i dont hav ea solution just reading code | 12:05 |
| *** shardy has joined #kolla | 12:05 | |
| *** mbound has quit IRC | 12:12 | |
| sdake | SamYaple what does this mean | 12:14 |
| sdake | TASK: [haproxy | Ensuring latest haproxy config is used] ********************** | 12:14 |
| sdake | failed: [minime-02] => {"changed": false, "cmd": ["docker", "exec", "haproxy", "/opt/kolla/ensure_latest_config.sh"], "delta": "0:00:00.013635", "end": "2015-09-28 12:14:40.150424", "rc": 255, "start": "2015-09-28 12:14:40.136789", "stdout_lines": [], "warnings": []} | 12:14 |
| sdake | FATAL: all hosts have already failed -- aborting | 12:14 |
| sdake | haproxy is in a restart loop | 12:17 |
| sdake | [ALERT] 270/121622 (7) : parsing [/etc/haproxy/haproxy.cfg:29] : 'bind' : invalid address: '192.1.68.1.148' in '192.1.68.1.148:3306' | 12:17 |
| sdake | oh i see the problem | 12:18 |
| sdake | nm | 12:18 |
| *** tfukushi_ has joined #kolla | 12:32 | |
| *** tfukushima has quit IRC | 12:34 | |
| *** gfidente has joined #kolla | 12:34 | |
| *** gfidente has joined #kolla | 12:34 | |
| sdake | samyaple what does this line do? : https://github.com/openstack/oslo.log/blob/master/oslo_log/log.py#L293 | 12:36 |
| SamYaple | it will call syslog.f() i believe | 12:38 |
| SamYaple | i have no idea why None is there | 12:38 |
| SamYaple | so it should call syslog.LOG_KERN() | 12:38 |
| SamYaple | " If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised." | 12:39 |
| SamYaple | so if LOG_KERN() doesn't exist, it returns None | 12:39 |
| SamYaple | well i guess it doesnt have to be a function | 12:39 |
| SamYaple | syslog.LOG_KERN | 12:39 |
| SamYaple | sdake: so all that is doing is checking what log facilties are available after the import https://docs.python.org/2/library/syslog.html | 12:41 |
| SamYaple | and if syslog wasn't imported successfully it wil lsay no log faciliets are vailable | 12:41 |
| SamYaple | so i bet syslog isn't being imported successfully | 12:42 |
| sdake | it seems like syslog isn't imported successfully | 12:42 |
| SamYaple | im going rpdb it | 12:42 |
| sdake | because in the type error, it prints out no log facilities | 12:42 |
| SamYaple | yes i think we have both come to this conclusion now | 12:42 |
| sdake | it just has an empty list | 12:42 |
| sdake | looking at oslo.log, I dont see why syslog wouldn't import successuflly | 12:43 |
| sdake | https://github.com/openstack/oslo.log/blob/master/oslo_log/log.py#L39 | 12:44 |
| SamYaple | it imports sucessfully for me | 12:44 |
| SamYaple | yea thats what i was saying earlier | 12:44 |
| SamYaple | if its none, then all things will be none | 12:45 |
| SamYaple | cause of the getattr | 12:45 |
| dimsum__ | not sure if this is related...https://review.openstack.org/#/c/227688/ | 12:45 |
| *** dimsum__ is now known as dims | 12:45 | |
| SamYaple | yea... thats right | 12:45 |
| SamYaple | i wonder how often the source binaries update | 12:45 |
| SamYaple | at tarballs.openstack | 12:46 |
| SamYaple | oh it doesn't matter since thats oslo.log | 12:46 |
| SamYaple | dims: thanks thats exactly it | 12:46 |
| SamYaple | sdake i suggest a patch | 12:46 |
| SamYaple | otherwise we have to wait for a new version of oslo.log to snap and requirements to be updated :D | 12:47 |
| * dims nods | 12:47 | |
| *** inc0 has joined #kolla | 12:48 | |
| *** inc0_ has joined #kolla | 12:51 | |
| *** itsuugo has quit IRC | 12:51 | |
| *** dwalsh_ has joined #kolla | 12:51 | |
| *** inc0 has quit IRC | 12:54 | |
| *** klint has quit IRC | 12:59 | |
| sdake | samyaple you suggest a patc hfor what | 12:59 |
| SamYaple | sdake: https://review.openstack.org/#/c/227688/ | 13:00 |
| SamYaple | like a .patch | 13:00 |
| sdake | why does this only happen in the neutron container? | 13:01 |
| SamYaple | im not seeing the other containers log either, but.... | 13:01 |
| sdake | so install oslo.log in openstack common container and patch it there? | 13:04 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Remove config-external for rsyslog https://review.openstack.org/228448 | 13:08 |
| SamYaple | sdake: that fixes the privelged thing for rsyslog | 13:08 |
| SamYaple | sdake: no because the other services seem fine... | 13:10 |
| SamYaple | going to check pip versions | 13:10 |
| sdake | Version: 1.11.1.dev2 | 13:10 |
| SamYaple | oslo.log==1.11.0 | 13:10 |
| SamYaple | thats keystone | 13:11 |
| SamYaple | oslo.log==1.11.0 | 13:11 |
| SamYaple | neutron | 13:11 |
| SamYaple | wierd | 13:11 |
| sdake | keystone = Version: 1.11.1.dev2 | 13:11 |
| sdake | with pip show | 13:11 |
| SamYaple | no thats the delorean stuff | 13:12 |
| SamYaple | but either way we both have the issue | 13:12 |
| inc0_ | I'm back | 13:12 |
| SamYaple | inc0_: https://review.openstack.org/228448 | 13:12 |
| SamYaple | that fixes the /dev/log creation | 13:12 |
| SamYaple | you need privelged to create teh device node | 13:12 |
| SamYaple | my container was borked | 13:12 |
| inc0_ | SamYaple, but you don't create /dev/log, you create /var/lib/kolla/dev | 13:13 |
| inc0_ | do you have correct perms on /var/lib/kolla? | 13:13 |
| SamYaple | inc0_: im jsut telling you what the logs say on a clean vanilla run | 13:14 |
| SamYaple | reruns work fine | 13:14 |
| inc0_ | probably that was the case | 13:14 |
| inc0_ | I'm not sure if privileged is required if you make /var/lib/kolla chmod 777 | 13:14 |
| SamYaple | which we arenlt going to do of course | 13:14 |
| inc0_ | (I did and it works for me) | 13:14 |
| inc0_ | yeah I guess | 13:15 |
| inc0_ | one way or another | 13:15 |
| *** dims_ has joined #kolla | 13:15 | |
| inc0_ | privileged is cleaner I guess | 13:15 |
| SamYaple | well be priv dropping in mitaka anyway | 13:15 |
| SamYaple | some containers will still need privleged, but they wont be running as root internally | 13:15 |
| SamYaple | so itll be fine | 13:15 |
| inc0_ | yeah, let's sort this out later then | 13:16 |
| SamYaple | inc0_: https://review.openstack.org/#/c/227900/ cliky the rebasey | 13:16 |
| SamYaple | or rebase. idk if web rebase will work | 13:17 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 13:17 |
| *** dims has quit IRC | 13:17 | |
| inc0_ | cliky the +2 SamYaple | 13:17 |
| SamYaple | already +1'd | 13:17 |
| inc0_ | thanks mate | 13:17 |
| SamYaple | anytime | 13:18 |
| SamYaple | sdake: https://bugs.launchpad.net/kolla/+bug/1489676 | 13:19 |
| openstack | Launchpad bug 1489676 in kolla "Leftover /var/run files prevent Keystone start" [High,Triaged] | 13:19 |
| SamYaple | I cant reproduce that with ubuntu even with your suggestions | 13:19 |
| sdake | ok | 13:20 |
| SamYaple | pbourke_: you around for a +2? all tested I promise 13:11 < SamYaple> thats keystone | 13:21 |
| SamYaple | https://review.openstack.org/#/c/228404/ | 13:21 |
| *** mbound has joined #kolla | 13:22 | |
| *** mbound has quit IRC | 13:22 | |
| *** mbound has joined #kolla | 13:23 | |
| *** tfukushi_ has quit IRC | 13:26 | |
| SamYaple | sdake: just applying the patch to neutron_server fixes it | 13:26 |
| SamYaple | no other containers have the issue wierd | 13:26 |
| sdake | we should probably apply globally i suspect | 13:27 |
| *** tfukushima has joined #kolla | 13:27 | |
| SamYaple | im cool with that ill work it up | 13:27 |
| sdake | cool I was working on it but if you want to tackle it wfm :) | 13:27 |
| sdake | i'll be happy to test centos bin and sourc ewhen your done | 13:28 |
| SamYaple | oh you can do it no problem | 13:28 |
| SamYaple | or i will no problem | 13:28 |
| SamYaple | tell me what to do | 13:28 |
| sdake | go ahead and do it | 13:28 |
| *** tfukushima has quit IRC | 13:28 | |
| sdake | and i'll test centos rdo and source | 13:28 |
| sdake | jus tto make sur eit works everywhere | 13:29 |
| openstackgerrit | Merged openstack/kolla: Rename ovs to openvswitch https://review.openstack.org/228404 | 13:29 |
| sdake | as long as you can do it shortly (by today) | 13:29 |
| *** tfukushima has joined #kolla | 13:31 | |
| *** rhallisey has joined #kolla | 13:33 | |
| rhallisey | morning | 13:33 |
| sdake | hey ryan | 13:34 |
| rhallisey | review q looks nice | 13:38 |
| SamYaple | rhallisey: we merrged like 30 patches this morning | 13:38 |
| rhallisey | nicce | 13:38 |
| rhallisey | I was getting my first cavity filled | 13:39 |
| rhallisey | my 23 year streak of no dental work ended today | 13:39 |
| *** jtriley has joined #kolla | 13:40 | |
| *** sdake_ has joined #kolla | 13:44 | |
| *** sdake has quit IRC | 13:47 | |
| *** achanda has joined #kolla | 13:49 | |
| *** sdake_ has quit IRC | 13:50 | |
| *** clayton has quit IRC | 13:51 | |
| *** achanda has quit IRC | 13:51 | |
| openstackgerrit | Merged openstack/kolla: Murano logging to rsyslog https://review.openstack.org/227900 | 13:54 |
| *** sdake has joined #kolla | 13:57 | |
| *** clayton has joined #kolla | 13:58 | |
| *** itsuugo has joined #kolla | 13:59 | |
| *** itsuugo has quit IRC | 14:01 | |
| SamYaple | sdake: ive got tied up i cant submit that to tonight for that patch | 14:01 |
| sdake | ok i'll do it then | 14:01 |
| sdake | thanks | 14:02 |
| jpeeler | when does rc-1 get cut? i just noticed ironic never made it in | 14:02 |
| sdake | jpeeler i am branching liberty and tagging rc1 tuesday 10am PST | 14:02 |
| sdake | recommend gettinga rebase, so we can merge your patch | 14:02 |
| jpeeler | yeah i'll look at it now | 14:03 |
| SamYaple | jpeeler: that ironic stuff won't work because of changes to config_external, but merge it anyway | 14:09 |
| SamYaple | ill submit the final json stuff for it | 14:09 |
| rhallisey | well it should still work | 14:10 |
| rhallisey | it just will need to be changed | 14:10 |
| SamYaple | rhallisey: no it won't work | 14:10 |
| SamYaple | rhallisey: https://review.openstack.org/#/c/228247/ | 14:10 |
| SamYaple | but ive been waiting for his ironic patch to merge to finish that out | 14:11 |
| SamYaple | plus i have to add pxe support for ubuntu | 14:11 |
| rhallisey | oh | 14:11 |
| *** absubram has quit IRC | 14:16 | |
| inc0_ | sdake, I don't think I'll make it with libvirt logging till tomorrow | 14:20 |
| inc0_ | or ovs | 14:20 |
| *** achanda has joined #kolla | 14:22 | |
| SamYaple | inc0_: ive got ovs and haproxy | 14:25 |
| SamYaple | i can look at the others tonight | 14:25 |
| SamYaple | ill have a few hours to do it | 14:25 |
| SamYaple | btw guys, did you notice how clean our start.sh got? and uniform? | 14:26 |
| inc0_ | well I'll look at it tomorrow | 14:26 |
| inc0_ | but I'm a bit afraid to rush it | 14:26 |
| inc0_ | if we break something, it will be severe | 14:26 |
| SamYaple | I am going to probe the waters on removing a specific start.sh per container but pushing in a generic start.sh in the base container and sourcing a specific (if needed) script later | 14:27 |
| SamYaple | so no more required start.sh per image | 14:27 |
| jpeeler | nice | 14:27 |
| SamYaple | no more duplicate code anywhere! | 14:27 |
| SamYaple | without symlinks! | 14:27 |
| inc0_ | SamYaple, sounds like good improvement for Mitaka:P | 14:27 |
| SamYaple | yea but im going to try to jam it in liberty anyway lol | 14:28 |
| openstackgerrit | Jeff Peeler proposed openstack/kolla: Add Ansible support for Ironic https://review.openstack.org/219747 | 14:28 |
| SamYaple | jpeeler: im pretty sure your containers for irnoic are broken, but they will still build and deploy (just crash) | 14:29 |
| SamYaple | that is ok ill fix in patch tonight, lets just get it merged. or you can fix now | 14:29 |
| jpeeler | Well... needs to be updated to use json config and what else? | 14:30 |
| SamYaple | jpeeler: thats it i believe, and the pxe for ubuntu | 14:31 |
| SamYaple | but id be ok with that missing rc1 | 14:31 |
| jpeeler | i can probably do the config file changes now | 14:32 |
| SamYaple | ok then the containers should launch if you do | 14:32 |
| *** dims has joined #kolla | 14:35 | |
| SamYaple | sdake: https://review.openstack.org/#/c/228448/ you can mark that off your list | 14:35 |
| *** dims_ has quit IRC | 14:35 | |
| inc0_ | SamYaple, all of CI builds failed there:S | 14:39 |
| inc0_ | also failed to build rsyslog | 14:40 |
| *** achanda has quit IRC | 14:40 | |
| sdake | samyaple INFO:__main__:base: ---> 991b8bb114a8 | 14:42 |
| sdake | INFO:__main__:base:Step 14 : RUN patch -p1 -d /usr/lib/python2.7/site-packages < /0001-Fix-unintended-assignment-of-syslog.patch | 14:42 |
| sdake | INFO:__main__:base: ---> Running in 031bcdad12ca | 14:42 |
| sdake | INFO:__main__:base:patching file oslo_log/log.py | 14:42 |
| sdake | INFO:__main__:base:Reversed (or previously applied) patch detected! Assume -R? [n] | 14:42 |
| sdake | INFO:__main__:base:Apply anyway? [n] | 14:42 |
| sdake | INFO:__main__:base:Skipping patch. | 14:42 |
| sdake | INFO:__main__:base:1 out of 1 hunk ignored -- saving rejects to file oslo_log/log.py.rej | 14:42 |
| sdake | INFO:__main__:base:Removing intermediate container 031bcdad12ca | 14:42 |
| sdake | so it looks like delorean has the patch applied already | 14:42 |
| *** diga has joined #kolla | 14:47 | |
| *** inc0_ has quit IRC | 14:53 | |
| *** absubram has joined #kolla | 14:59 | |
| *** banix_ has joined #kolla | 15:05 | |
| *** jtriley has quit IRC | 15:07 | |
| sdake | SamYaple ping? | 15:16 |
| *** jasonamyers has joined #kolla | 15:22 | |
| *** jtriley has joined #kolla | 15:23 | |
| *** banix_ is now known as banix | 15:24 | |
| *** absubram has quit IRC | 15:25 | |
| *** inc0 has joined #kolla | 15:27 | |
| SamYaple | sdake: pong | 15:38 |
| SamYaple | about to sleep though | 15:38 |
| sdake | hey sam | 15:38 |
| sdake | so not sure why patch is saying the patch is already applied | 15:39 |
| sdake | but it is clearly not | 15:39 |
| sdake | some wierdness going on | 15:39 |
| sdake | but i'll sort it out, sleep well :) | 15:39 |
| SamYaple | ok feel free to submit a review and ill update it tonight if its still broke | 15:39 |
| SamYaple | then test the agents cause they were busted too | 15:39 |
| sdake | ya feel free to hack on my change if you dont like it | 15:39 |
| sdake | goal is to get it in fast | 15:39 |
| SamYaple | but ill work on it tonight to get it all in tiptop shape | 15:40 |
| sdake | i'll try to test deploy of ubuntu when i get it in working order | 15:41 |
| *** mat425 has joined #kolla | 15:41 | |
| SamYaple | thanks sdake | 15:42 |
| sdake | priority #1 for me atm | 15:42 |
| *** inc0 has quit IRC | 15:45 | |
| mat425 | Hi all guys, i'm trying kolla, but i'm instantly sucked with an issue :(. For dev purpose i tried kolla using Vagrant, so i basically followed the steps on github. but once vagrant up command arrives on "Configuring and enabling network interfaces" it fails, the error now is "No guest IP was given to the Vagrant core NFS helper. This is an internal error that should be reported as a bug." but before i got a generic one | 15:46 |
| *** dims has quit IRC | 15:46 | |
| *** dims has joined #kolla | 15:46 | |
| sdake | mat425 the person your looking for is mandre, but he is asleep atm I think | 15:46 |
| sdake | you ahve to catch him about 5 hours earlier | 15:46 |
| SamYaple | pbourke_ may be able to help | 15:47 |
| *** inc0 has joined #kolla | 15:47 | |
| sdake | mat425 i'd highly recommend just doing an AIO deploy on vm or bare metal to get started | 15:47 |
| sdake | rather then use vagrant | 15:47 |
| sdake | our vagrant dev env is a little bit pre-alpha quality ;) | 15:47 |
| sdake | it just went through a major refactor to add libvirt support | 15:47 |
| sdake | your running on linux, correct? | 15:47 |
| SamYaple | i would say its solid beta | 15:47 |
| sdake | ok solid beta | 15:47 |
| mat425 | ah okok got it | 15:47 |
| sdake | still not rockin quality like the rest of kolla | 15:48 |
| mat425 | yes running linux, ubuntu 14.04 | 15:48 |
| sdake | which is deployable now | 15:48 |
| sdake | mat425 i haven't tried out the new vagrant work which added the libvirt port | 15:48 |
| sdake | so I don't know how to help you | 15:48 |
| mat425 | i had to install nfs and libvirt even though i've not really got why | 15:48 |
| pbourke_ | mat425: you'd be better using virtualbox imo | 15:48 |
| sdake | pbourke_ could possibly help too, but he is probably gone (2 hours earlier yous hould be able to catch him) | 15:48 |
| sdake | mat425 are you on vbox? | 15:48 |
| mat425 | vagrant with virtualbox yes | 15:49 |
| sdake | ahh, well pbourke is your man then :) he knows that part of the system well | 15:49 |
| pbourke_ | mat425: change this line to "virtualbox" https://github.com/openstack/kolla/blob/master/vagrant/Vagrantfile#L7 | 15:49 |
| mat425 | i was trying vagrant just for a fast deployment, i have to deploy it on a cluster later on | 15:49 |
| sdake | ya vagrant is good for eval | 15:50 |
| pbourke_ | and if you want multinode change this line to true https://github.com/openstack/kolla/blob/master/vagrant/Vagrantfile#L25 | 15:50 |
| sdake | just not sure what state its in currently because I haven't personally tested it | 15:50 |
| pbourke_ | then tell tell everyone how I was right in that virtualbox should be the defualt ;) | 15:50 |
| sdake | pbourke fwiw I can't get vbox to run on fedora | 15:50 |
| sdake | but i dont mind changing the default back to vbox | 15:51 |
| pbourke_ | ah I dont want to go down that road again | 15:51 |
| sdake | was that issue never settled? | 15:51 |
| pbourke_ | we never put it to a vote | 15:51 |
| sdake | sorry, I've been mia for a bit, been down with back to back flu/cold | 15:51 |
| sdake | did we need to vote on it? | 15:51 |
| sdake | i'll kick that off if there is a logjam on the issue | 15:51 |
| mat425 | ok good, i'm gonna try it right now, but anyway it's ok for me to install it on a clean new virtualbox | 15:51 |
| sdake | i'd rather not havea vote on every little detail but this is a big detail | 15:52 |
| mat425 | (still with vagrant) | 15:52 |
| pbourke_ | yeah I think its divided enough that a vote would help. its not a big deal either way but when I see newcomers getting stuck... | 15:52 |
| mat425 | by the way thanks | 15:52 |
| pbourke_ | its like one less thing for them to trip up on | 15:52 |
| sdake | pbourke is it not documented how to set vbox vs libvirt? | 15:53 |
| mat425 | actually it was really clear how to do it | 15:54 |
| sdake | pbourke beyond the deafult setting, was there any other issue that needed community agreement? | 15:54 |
| sdake | or gaps in the implementation choices? | 15:54 |
| pbourke_ | sdake: not that Im aware of | 15:54 |
| sdake | (so I can craft the email) | 15:55 |
| sdake | so its just the "what hsould the default setting be"? | 15:55 |
| pbourke_ | yes | 15:55 |
| sdake | roger i'll send an email out today | 15:55 |
| pbourke_ | we're not going to fall out over it but would be good to put it to bed once and for all I think | 15:55 |
| pbourke_ | thx | 15:55 |
| sdake | agreed, we use votes to solve logjams about policy | 15:56 |
| sdake | if tis a technical issue, we should just be able to fix that with patches | 15:56 |
| sdake | but this sounds like a policy issue (defaults are policy issues) | 15:56 |
| sdake | mat425 note if your lauching your vms inside a vm, you need to add an augmentation blob | 15:57 |
| sdake | its in the documentation | 15:57 |
| mat425 | another question, do you think that, once deployed, adding extra modules is still that easy? (i need also to install Monasca and Openstack-Neat) | 15:57 |
| sdake | (launching vms ifrom nova inside a vm) | 15:57 |
| sdake | so for monasca, we would probably want to add a container set for that and ansible playbook stuff for it | 15:58 |
| sdake | i am not familiar with openstack-neat | 15:58 |
| sdake | but we dont' typically containerize stuff that isn't under big tent governance | 15:58 |
| sdake | although if someone submits code, we will merge it | 15:58 |
| sdake | as for doing the work, its "easy" for someone experienced inthe process and hard for someone not :) | 15:58 |
| pbourke_ | to answer the question though, I dont think there should be anything stopping you adding these once deployed | 15:59 |
| sdake | right | 15:59 |
| pbourke_ | even if outside containers | 15:59 |
| sdake | it would probably take a cor ereviever about 8-16 hours to add a new service | 15:59 |
| sdake | wow, 301 plays | 15:59 |
| sdake | kolla getting busy :) | 15:59 |
| mat425 | ok thank you | 15:59 |
| sdake | and we are totally open to accepting these features into master | 16:00 |
| sdake | (if you do the work for example) | 16:00 |
| sdake | our objective is to deploy the big tent in containers | 16:00 |
| sdake | mat425 we really want to make sure you have a good experience in your deployment, so feel free to hit up the devs with your questions | 16:02 |
| sdake | mat425 may I ask - you mentioned youw ere goin to do a deployment later, how many nodes? | 16:02 |
| mat425 | thanks! i've already asked 2 questions here and the answers were fast and complete, i totally gonna hit this chat :) | 16:04 |
| mat425 | i'm working on my thesis (MSc), at uni we have a 4 machine cluster | 16:05 |
| mat425 | a little one | 16:05 |
| sdake | cool | 16:05 |
| sdake | well kolla will defiantely work for that envrionment | 16:05 |
| sdake | over 100 nodes and i'm not clear how well things would work out of the box | 16:05 |
| mat425 | Maybe (with a big M) i can test my work on a bigger cluster at hp | 16:05 |
| sdake | (which is why I asked) | 16:05 |
| mat425 | ok | 16:05 |
| mat425 | i had a look also to the multinode set up (still in vagrant), don't really get why a cluster of 5 nodes needs 3 controller | 16:06 |
| mat425 | :\ | 16:06 |
| pbourke_ | mat425: I've been meaning to change it to 2 | 16:09 |
| *** blahRus has joined #kolla | 16:09 | |
| pbourke_ | mat425: 2 is min for HA | 16:09 |
| mat425 | what is HA? | 16:10 |
| *** inc0 has quit IRC | 16:10 | |
| pbourke_ | high availability | 16:12 |
| mat425 | ah yeah sure | 16:12 |
| mat425 | o | 16:12 |
| mat425 | ok | 16:12 |
| *** cloudnau_ has joined #kolla | 16:16 | |
| *** unicell has quit IRC | 16:17 | |
| *** inc0 has joined #kolla | 16:35 | |
| *** Kennan2 has joined #kolla | 16:36 | |
| *** Kennan has quit IRC | 16:36 | |
| sdake | 3 controllers are needed pbourke for ha | 16:37 |
| sdake | 2 wont do it | 16:37 |
| sdake | pbourke_ ^^ | 16:37 |
| pbourke_ | oh | 16:38 |
| pbourke_ | is that because of galera? | 16:38 |
| sdake | i think so | 16:38 |
| openstackgerrit | Jeff Peeler proposed openstack/kolla: Add Ansible support for Ironic https://review.openstack.org/219747 | 16:38 |
| sdake | i could be mistaken | 16:38 |
| *** achanda has joined #kolla | 16:40 | |
| pbourke_ | ok good to know | 16:43 |
| sdake | sam would know for certain | 16:43 |
| sdake | everyone tells me 3 nodes | 16:43 |
| sdake | so I just paroot :) | 16:43 |
| sdake | parrot | 16:43 |
| *** tfukushima has quit IRC | 16:43 | |
| inc0 | yeah it's for galera | 16:43 |
| inc0 | either 3 or 5...odd number anyway | 16:43 |
| inc0 | sdake, you of all people should know mr. corosync;) | 16:44 |
| sdake | corosync doesn't car eabout quorum | 16:44 |
| sdake | EVS doesn't have any notion of quorum :) | 16:44 |
| sdake | EVS is designed to split brain | 16:44 |
| sdake | how proper distributed systems should be made imo | 16:44 |
| sdake | but nobody can work with it because it requires more brain power to iplement systems on top of it :) | 16:45 |
| inc0 | same goes for riak as far as I know | 16:45 |
| inc0 | it behaves very well in terms of HA, but it's so twisted noone knows how to use it | 16:46 |
| *** sdake_ has joined #kolla | 16:46 | |
| *** sdake has quit IRC | 16:49 | |
| *** achanda has quit IRC | 16:54 | |
| *** unicell has joined #kolla | 16:58 | |
| *** inc0 has quit IRC | 17:06 | |
| *** mbound has quit IRC | 17:07 | |
| *** rhallisey has quit IRC | 17:34 | |
| *** athomas has quit IRC | 17:48 | |
| *** achanda has joined #kolla | 17:54 | |
| *** loth has joined #kolla | 17:56 | |
| *** diga has quit IRC | 17:58 | |
| *** dolpher has joined #kolla | 17:59 | |
| *** achanda has quit IRC | 17:59 | |
| *** athomas has joined #kolla | 18:00 | |
| *** mbound has joined #kolla | 18:07 | |
| *** mbound has quit IRC | 18:13 | |
| *** mbound has joined #kolla | 18:18 | |
| openstackgerrit | Jeff Peeler proposed openstack/kolla: Add Ansible support for Ironic https://review.openstack.org/219747 | 18:23 |
| *** sdake has joined #kolla | 18:28 | |
| *** sdake_ has quit IRC | 18:31 | |
| sdake | jpeeler is rhallisey about? | 18:33 |
| sdake | I have an emergency with the neutron agents being busted | 18:33 |
| sdake | that is a result of the json config changes | 18:33 |
| *** akwasnie has joined #kolla | 18:40 | |
| *** jasonamyers has quit IRC | 18:47 | |
| *** achanda has joined #kolla | 18:56 | |
| *** achanda has quit IRC | 19:00 | |
| jpeeler | he appears to be gone | 19:01 |
| jpeeler | ah he just came back | 19:01 |
| *** rhallisey has joined #kolla | 19:01 | |
| jpeeler | rhallisey: sdake was looking for you 30 minutes ago. neutron agents json config related | 19:02 |
| sdake | rhallisey this is blocking the release of rc1: https://bugs.launchpad.net/kolla/+bug/1500566 | 19:02 |
| openstack | Launchpad bug 1500566 in kolla "neutron_agents is broken with set_configs JSON" [Critical,Confirmed] - Assigned to Sam Yaple (s8m) | 19:02 |
| sdake | can you fix it please | 19:02 |
| sdake | i want to tag and branch asap | 19:02 |
| sdake | (10am tomorrow) | 19:02 |
| *** banix has quit IRC | 19:03 | |
| rhallisey | what specifically is broekn | 19:03 |
| rhallisey | oh I see the paste one second | 19:03 |
| rhallisey | oh I didn't write a patch for this one | 19:04 |
| rhallisey | since there are like 4 services in there | 19:04 |
| rhallisey | I'll take a look though | 19:04 |
| rhallisey | sam must've patched it | 19:05 |
| rhallisey | I didn't think the new config stuff would work because there are lik 4 services | 19:05 |
| *** athomas has quit IRC | 19:06 | |
| rhallisey | oh actually | 19:08 |
| sdake | jpeeler really nice job on ironic | 19:08 |
| rhallisey | I think I know.. | 19:08 |
| sdake | really hard technical work - hope it was rewrading :) | 19:08 |
| sdake | well its busted | 19:08 |
| sdake | it needs to be fixed ;) | 19:08 |
| jpeeler | sdake: thanks, but it's sort of limited as you know and a bit rushed | 19:09 |
| sdake | yup | 19:09 |
| jpeeler | hopefully can get it fully good next release cycle | 19:09 |
| sdake | it is limited | 19:09 |
| sdake | but we will make it rockstar status in the next release :) | 19:09 |
| sdake | it is till really nicely done | 19:09 |
| sdake | given the limitations of ironic | 19:09 |
| *** itsuugo has joined #kolla | 19:11 | |
| sdake | jpeeler isn't pxe setup for ubuntu same as for rpm distros? | 19:12 |
| rhallisey | sdake, we're going to break apart neutron_agents in the future right? | 19:12 |
| jpeeler | sdake: actually i think it's slightly more complicated. the files need some copying as the packages are different (i think) | 19:13 |
| sdake | it doesn't matter what we do in the future, we are releasing rc1 based upon a fat container | 19:13 |
| sdake | jpeeler ok | 19:13 |
| sdake | rhallisey please review https://review.openstack.org/#/c/219747/ | 19:14 |
| sdake | needs to merge | 19:14 |
| sdake | unless there is some critical problem | 19:14 |
| sdake | jpeeler please file a tech debt bug and assign to rc1 for the pxe + ubuntu stuff | 19:14 |
| sdake | if you dont' want sam busting your implementation, i'd suggest adding the json config stuff in after this patch hits the repo jpeeler | 19:15 |
| jpeeler | already did :) | 19:15 |
| *** dolpher has quit IRC | 19:16 | |
| rhallisey | sdake, the way the json stuff works will require not having a fat container. But I can just add back in the config-external bits just for neutron agents. Unless I figure out a way around it. | 19:16 |
| sdake | wrong answer | 19:16 |
| sdake | both of those are wrong answers :) | 19:17 |
| sdake | make it work with json | 19:17 |
| sdake | just refactor set_config | 19:17 |
| sdake | config-external is gone | 19:17 |
| sdake | sam removed he infrastructure ot support it | 19:17 |
| *** marun has joined #kolla | 19:17 | |
| sdake | ironic is probablybroken at this moment as well | 19:17 |
| rhallisey | it would be an easy fix. Just put it into start.s | 19:17 |
| rhallisey | start.sh | 19:17 |
| marun | I'm working on deploying just neutron with kolla | 19:18 |
| rhallisey | I would be changing set_configs for something that will go away in the future | 19:18 |
| jpeeler | sdake: i updated the ironic patch, it should be completely on par as before | 19:18 |
| marun | I'm seeeing a hard vm crash when ovs containers are started. | 19:18 |
| sdake | jpeeler do you use the new json config method? | 19:18 |
| jpeeler | sdake: yes | 19:18 |
| rhallisey | I don't want to refactor the python script to account for something that we are fixing later | 19:18 |
| marun | Has anyone else seen this? | 19:18 |
| sdake | marun have never seen that, but atm neutron_agents is completely borked | 19:19 |
| sdake | i really wish people woudl *TEST* their changes | 19:19 |
| sdake | ok rhallisey | 19:19 |
| sdake | what do you propose we do? | 19:19 |
| sdake | not release rc1? | 19:19 |
| rhallisey | no your misunderstanding. I skipped this piece because I knew it wouldn't work | 19:20 |
| sdake | can you explain why it can't work? | 19:20 |
| rhallisey | let me just write a patch for it | 19:20 |
| sdake | it could be made to work | 19:20 |
| rhallisey | because we have 4 services and the python script looks for 1 config,json | 19:20 |
| sdake | intel 750 PCIE SSD = lifesaver | 19:20 |
| sdake | can't you just hae the 1 config json for al four services anyway? | 19:21 |
| rhallisey | ya I could, but there would also be 4 commands in there | 19:21 |
| sdake | and ahve it copy over a directoyr structure or something? | 19:21 |
| sdake | i see, so the problem is the commands not hte copy files | 19:21 |
| rhallisey | I can do the copy | 19:21 |
| rhallisey | then hard code the commands in start.sh | 19:22 |
| sdake | ok lets do that | 19:22 |
| rhallisey | yes axactly | 19:22 |
| sdake | cool winnign :) | 19:22 |
| rhallisey | ya no problem. I was aware of this issue when going through | 19:22 |
| sdake | thanks for helping me throug hthat | 19:22 |
| sdake | sorry if i seem shsort, feeling sickly | 19:22 |
| rhallisey | no worries | 19:22 |
| sdake | ya I think sam wasn't when he completely removed config-external infrastructure support in kolla-common.sh | 19:22 |
| rhallisey | I forgot to tell sam this when he ripped out config-external | 19:22 |
| rhallisey | I was going to do that part but sam kinda just jumped in | 19:23 |
| sdake | ya he does that :) | 19:23 |
| sdake | that is why he gets paid the big bucks | 19:23 |
| sdake | or something :) | 19:23 |
| sdake | removing config-external is good | 19:23 |
| sdake | a hardcode in this case is the right answer | 19:24 |
| sdake | jpeeler can you write a tech debt bug for the pxe + ubuntu support | 19:24 |
| jpeeler | https://bugs.launchpad.net/kolla/+bug/1500577 | 19:24 |
| openstack | Launchpad bug 1500577 in kolla "ironic needs discoverd service for source and PXE service for deb binary" [Undecided,New] - Assigned to Sam Yaple (s8m) | 19:24 |
| sdake | rhallisey can you ack that ironic patch | 19:24 |
| sdake | marun so, once rhallisey fixes this problem, I'd suggest trying his change | 19:25 |
| marun | sdake: fix for neutron agents? | 19:25 |
| rhallisey | I'll review in a moment | 19:25 |
| sdake | but another possiblity is your neutron network and management network are not set properly in your globals.yml | 19:25 |
| sdake | marun ack | 19:25 |
| rhallisey | let me just push this. Should take a second | 19:25 |
| marun | sdake: I'm not sure it's a networking issue. my vm literally dies | 19:25 |
| sdake | you mean the qemu process exits? | 19:26 |
| marun | sdake: vmware fusion on os x | 19:26 |
| marun | sdake: but same thing, yeah | 19:26 |
| marun | sdake: although.... | 19:26 |
| marun | sdake: maybe it's just losing connectivity :/ | 19:27 |
| sdake | so your running kolla within a vm and deploy a vm ? | 19:27 |
| sdake | marun right that is my suspicion, your networks are not setup properly | 19:27 |
| sdake | the neutron network has to not be your management network | 19:27 |
| sdake | if it is, it will lose connectivity to the vm | 19:27 |
| *** banix has joined #kolla | 19:28 | |
| marun | sdake: looks like vm actually dies, actually | 19:28 |
| sdake | hrm | 19:29 |
| sdake | did you override libvirt to use qemu instead of kvm for hypervisoring? | 19:29 |
| marun | sdake: not deploying kvm | 19:29 |
| *** erkules_ has joined #kolla | 19:29 | |
| sdake | your booting with nova yes? | 19:29 |
| marun | sdake: No | 19:29 |
| sdake | ok, confused :) | 19:30 |
| marun | sdake: my intention is to be able to deploy neutron with kolla, and just neutron. | 19:30 |
| marun | sdake: the first part is getting things deploying | 19:30 |
| sdake | i heard he part about your only using ovs | 19:30 |
| sdake | quick Q< why do you want to do that? | 19:30 |
| marun | sdake: the second part is to get a fake hypervisor working, think container running docker-in-docker instead of kvm | 19:30 |
| sdake | just wondering on the use case | 19:30 |
| *** erkules has quit IRC | 19:31 | |
| marun | sdake: neutron development shouldn't require any other openstack service | 19:31 |
| marun | sdake: it adds unnecessary complexity | 19:31 |
| sdake | is that maru newby? | 19:31 |
| marun | yes | 19:31 |
| sdake | hey o/ | 19:31 |
| marun | sdake: I'm helping with openshift on openstack integration, and having to work with a full openstack deployment isn't really necessary for most of the work | 19:31 |
| sdake | ok, well i'm not totally sure about how to make your use cae work, but i'd start with getting a full stack operational in your fusion hypervisor | 19:32 |
| sdake | ocne that works, you can trim the fat | 19:32 |
| sdake | does that make sense? | 19:32 |
| sdake | i dont know if anyone has ever deployed on fusion before | 19:32 |
| sdake | most people on osx use vbox | 19:32 |
| sdake | or all :) | 19:32 |
| marun | sdake: Maybe I should switch to vbox | 19:33 |
| marun | sdake: since I'm not doing nested virt | 19:33 |
| sdake | worth a go | 19:33 |
| sdake | even fi you do nested virt, you can run qemu for the virt | 19:33 |
| marun | sdake: fusion on osx is the only way to have accelerated nested virt. vbox doesn't support | 19:33 |
| sdake | i'd really recommend tryign to get a full stack going | 19:33 |
| sdake | oh wasn't aware of that | 19:33 |
| marun | sdake: but as you say, qemu still works | 19:33 |
| sdake | that way you know you have something working in th kolla frame | 19:33 |
| sdake | and then once that works, you can try to trim nova | 19:34 |
| sdake | you should be asble to do so | 19:34 |
| sdake | turn off all services | 19:34 |
| marun | sdake: I'm deploying with --tags rabbitmq,mariadb,neutron-only | 19:34 |
| marun | sdake: where neutron-only doesn't pull in compute | 19:34 |
| sdake | ya that wont work | 19:34 |
| sdake | you ned haproxy and keepalived | 19:34 |
| marun | sdake: really? | 19:34 |
| sdake | really | 19:34 |
| sdake | you can make it work but it requires some special magic | 19:34 |
| sdake | again, your tyring to optimize too many steps at once | 19:35 |
| marun | sdake: trying to keep things simple :) | 19:35 |
| sdake | big, smaller, smaller, smaller etc | 19:35 |
| *** gfidente has quit IRC | 19:35 | |
| sdake | your goal is a simple system but you are taking complex steps to achcieve it :) | 19:35 |
| marun | sdake: why are haproxy and keepalived required? | 19:35 |
| sdake | takesimple steps to achieve it | 19:35 |
| sdake | everything goes through haproxy | 19:35 |
| sdake | keepalived assigns the ip for haproxy | 19:35 |
| marun | sdake: ok | 19:36 |
| sdake | you can make kolla work witthout it | 19:36 |
| sdake | but I don't have a howto on the subject | 19:36 |
| sdake | again, baby steps :) | 19:36 |
| marun | fair enough | 19:36 |
| marun | regarding deployment, is it possible to deploy to the same host (devstack-style) or do I need a separate host to target? | 19:37 |
| marun | I can see how network munging on the deployment host could get messsy | 19:37 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Make oslo.log work properly with syslog https://review.openstack.org/228595 | 19:38 |
| sdake | same host | 19:38 |
| sdake | or different host | 19:38 |
| sdake | both work | 19:38 |
| sdake | i do both in my environment | 19:38 |
| sdake | when i do same host, its AIO testing | 19:38 |
| sdake | when I do different hosts, its 3 bare metal HA setup | 19:38 |
| marun | ok, good to know | 19:39 |
| marun | and thanks for the pointer re: intel 750 | 19:39 |
| sdake | ya, rockin hardware | 19:40 |
| marun | I've been meaning to rebuild a dev host, but it's only real advantage over mac with pcie is 32gb ram. having significantly faster io is a good excuse to go back to developing off my laptop | 19:40 |
| sdake | $385 | 19:40 |
| sdake | i have 128gb ram | 19:40 |
| sdake | and 3 32gb ram bare metal machines as well | 19:41 |
| marun | that's some nice iron | 19:41 |
| sdake | 128gb ram 32 cores (16 real, 16 hyperthreaded) | 19:41 |
| sdake | hp z820 | 19:41 |
| sdake | itspretty nice gear :) | 19:41 |
| sdake | the other machcines are minidell 710s I htink | 19:41 |
| marun | I'm not really working on openstack anymore, though, and more modest hardware works nicely for container workloads :) | 19:42 |
| sdake | marun what are you working on now then? | 19:44 |
| sdake | jpeeler rhallisey this is a new patcch but the openstack bot didn't show it | 19:44 |
| sdake | https://review.openstack.org/#/c/228595/ | 19:44 |
| marun | sdake: project atomic/openshift | 19:45 |
| sdake | dontt report to pmyers anymore? | 19:45 |
| marun | nope | 19:45 |
| sdake | interesitng | 19:45 |
| marun | not that I've managed to entirely escape, openstack integration is a hot topic | 19:46 |
| *** dolpher has joined #kolla | 19:47 | |
| *** achanda has joined #kolla | 19:56 | |
| *** achanda has quit IRC | 20:01 | |
| sdake | jpeeler I think your ironic patch breaks the gates https://review.openstack.org/#/c/219747/ | 20:04 |
| sdake | ya I bet :) | 20:04 |
| jpeeler | sdake: it's not actually broken, it's just reporting that source isn't implemented, | 20:06 |
| sdake | jpeeler i -1 that ironic patch - it is definately an ironic patch that break it | 20:06 |
| sdake | cool so override the tet patch so it doesn't return failure pls :) | 20:06 |
| sdake | test patch | 20:06 |
| sdake | I want people to trust the gate | 20:07 |
| jpeeler | i got -1 reviews for not putting them in there | 20:07 |
| sdake | and we can't do that if it is broken | 20:07 |
| sdake | we just fixed it htis morning | 20:07 |
| sdake | what I mean is thetest_build.py or whatver its called | 20:07 |
| sdake | has a blacklist or whitelist or whatever | 20:07 |
| sdake | add it there | 20:07 |
| jpeeler | ah | 20:07 |
| jpeeler | ok, though i just realized that script doesn't support differences between types | 20:08 |
| sdake | yup | 20:08 |
| sdake | I hvae a bug for that in the tracker | 20:08 |
| sdake | a whitelist for source and whitelist for binary | 20:08 |
| sdake | feel free to fix that too - its in rc2 ;) | 20:08 |
| sdake | but if not, just whitelist it entirely I think it shouldbe ok | 20:08 |
| *** dolpher has quit IRC | 20:08 | |
| sdake | it works for murano and gnocchi i think | 20:08 |
| jpeeler | yeah, i'm sure it'll make it green | 20:09 |
| sdake | as long as it passes | 20:09 |
| sdake | I want the gate to be trusted :) | 20:09 |
| sdake | it was trusted for a long time and someone broke the gate | 20:09 |
| sdake | and its taken like 2 weeks to fix it | 20:09 |
| sdake | I don't even know what patch broke it | 20:09 |
| sdake | 11 blueprints and 87 bugs, not bad output for only 3 weeks ;) | 20:10 |
| openstackgerrit | Jeff Peeler proposed openstack/kolla: Add Ansible support for Ironic https://review.openstack.org/219747 | 20:10 |
| sdake | rhallisey is that patch coming along? | 20:10 |
| sdake | if you don't finish, please git review it so same can take a look | 20:10 |
| sdake | rhallisey ^^ | 20:11 |
| sdake | same/sam | 20:11 |
| sdake | marun who is your manager now? | 20:11 |
| marun | sdake: jack rieden | 20:13 |
| sdake | heard he is a ogod guy | 20:13 |
| sdake | jpeeler I don't understand your request on the oslo.log patch | 20:13 |
| sdake | jpeeler yese patch should be short lived | 20:13 |
| jpeeler | just wanted to make it clear to not just remove the patch, but to remove the patch packages too | 20:14 |
| sdake | there is a big TODO that the stuff needs to be removed | 20:14 |
| sdake | which patch packages? | 20:14 |
| sdake | you mean 0001-Fix-*patch? | 20:14 |
| sdake | do you mean the patch files ? | 20:14 |
| jpeeler | the yum install patch, and apt-get install patch | 20:14 |
| sdake | oh right | 20:14 |
| sdake | thanks | 20:14 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Make oslo.log work properly with syslog https://review.openstack.org/228595 | 20:15 |
| sdake | try now :) | 20:15 |
| sdake | rhallisey ^^ please review that change | 20:16 |
| *** shardy has quit IRC | 20:17 | |
| *** sdake_ has joined #kolla | 20:25 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Neutron Agents json support https://review.openstack.org/228611 | 20:25 |
| rhallisey | sdake, ^ try that | 20:25 |
| rhallisey | you should be able to build that 10x faster then me | 20:25 |
| *** sdake has quit IRC | 20:28 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Neutron Agents json support https://review.openstack.org/228611 | 20:29 |
| rhallisey | was missing a comma | 20:29 |
| rhallisey | sdake_, oh by the way, I know why cinder doesn't work | 20:30 |
| sdake_ | and taht is what? | 20:30 |
| rhallisey | remember that cinder fix I did that added tgt back in? For some reason tgt is no longer in cinder | 20:30 |
| rhallisey | no idea how | 20:31 |
| *** jtriley has quit IRC | 20:34 | |
| sdake_ | rhallisey sincey ou fixed the bug please assign it to yourself thnaks :) | 20:34 |
| rhallisey | I did | 20:35 |
| rhallisey | can yo help me confirm | 20:35 |
| rhallisey | your machine is like 100 times faster than mine | 20:35 |
| sdake_ | wasn't the neutron agents the entire reason we did the json copy thing? | 20:37 |
| sdake_ | you mean you didn't test it? | 20:37 |
| rhallisey | no not neutron-agents | 20:38 |
| * sdake_ groans | 20:38 | |
| sdake_ | how do you know it works? | 20:38 |
| *** sdake_ is now known as sdake | 20:38 | |
| rhallisey | it was for neutron-server and ovs-agent | 20:38 |
| *** dolpher has joined #kolla | 20:39 | |
| rhallisey | sdake, the container shouldn't exit | 20:39 |
| *** achanda has joined #kolla | 20:39 | |
| rhallisey | the logs should show the configs being copied | 20:39 |
| rhallisey | sdake, I'm testing it. It just is going to take an hour to build this | 20:40 |
| rhallisey | your fancy computer takes like 30s | 20:40 |
| *** sdake has quit IRC | 20:42 | |
| *** dolpher has quit IRC | 20:43 | |
| *** sdake has joined #kolla | 20:43 | |
| sdake | rhallisey please ack https://review.openstack.org/#/c/228595/ | 20:43 |
| sdake | rhallisey please review https://review.openstack.org/#/c/219747/ | 20:43 |
| sdake | we are almost through the home stretch of rc1 :) | 20:43 |
| rhallisey | nice! Looking at ironic.. | 20:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove config-external for rsyslog https://review.openstack.org/228448 | 20:46 |
| sdake | merge that oslo.log patch first | 20:47 |
| sdake | its easy peasy to review | 20:47 |
| sdake | rhallisey ^^ | 20:47 |
| openstackgerrit | Merged openstack/kolla: Make oslo.log work properly with syslog https://review.openstack.org/228595 | 20:50 |
| sdake | rhallisey ta | 20:50 |
| sdake | rhallisey btw the reason I have smoking fast hardware is so I can test *my* changes faster, not be the single point of testing for everyone | 20:53 |
| rhallisey | sdake, ya I understand. I figured you had an env ready to go to test it | 20:54 |
| rhallisey | all you need to do it build and run | 20:54 |
| rhallisey | but I don't care I'll do it | 20:54 |
| sdake | same as you right? :) | 20:54 |
| sdake | i've got it running in a moment | 20:54 |
| sdake | but if its busted i'm not debugging it :) | 20:54 |
| sdake | doesn't work | 20:54 |
| rhallisey | I know it's the same :P. Mines just going to take longer so if you want to wait idc | 20:54 |
| rhallisey | what do the logs get? | 20:55 |
| sdake | hang tight | 20:55 |
| rhallisey | kk | 20:55 |
| rhallisey | thanks | 20:55 |
| sdake | btw, all developers should be testing their changes | 20:55 |
| sdake | i am getting alot of private messages from other devs annoyed that people don't test their changes | 20:55 |
| sdake | it pisses everyone off whne shit gets broken constantlly | 20:55 |
| sdake | part of the problem is we have no automated functionalgate | 20:55 |
| sdake | which we need to solve | 20:55 |
| sdake | but part of the commit process is testing the change | 20:56 |
| sdake | I realize its onerous to test changes because the build and deploy takes 45 minutes | 20:56 |
| sdake | but its onerous on other developers to bisect a busted commit | 20:56 |
| rhallisey | if I don't test it I'll just -1 it | 20:56 |
| rhallisey | agreeed | 20:56 |
| rhallisey | didn't* | 20:57 |
| sdake | rhallisey http://ur1.ca/nvi3n -> http://paste.fedoraproject.org/272382/73882144 | 20:58 |
| sdake | don't submit changes without commit unless its end of day | 20:58 |
| sdake | please | 20:58 |
| sdake | without tewswting i mean | 20:58 |
| rhallisey | can I -1 if I didn't test though? | 20:59 |
| sdake | how is that helpful unless its end of day and your doing a handoff? | 21:00 |
| rhallisey | just to receive comments | 21:00 |
| sdake | i see, well we aren't after comments here, we are after a working solution :) | 21:00 |
| sdake | i want to pull the trigger on this release tomorrow morning | 21:00 |
| rhallisey | in this case it was so you could help me test :) | 21:01 |
| sdake | yes lazy lazy | 21:01 |
| rhallisey | but I'm built now so I should be good | 21:01 |
| sdake | it frustrates other developers | 21:01 |
| sdake | including me :) | 21:01 |
| rhallisey | I blame my hardware | 21:01 |
| sdake | i tell ya, that intel 750 drastically ahs improved my well being | 21:01 |
| rhallisey | sam said he will help me though | 21:01 |
| sdake | ya | 21:02 |
| sdake | but if youcan get it done plase do | 21:02 |
| sdake | i'd like to be able to test for sevearl hours tonight | 21:02 |
| rhallisey | shouldn't be a problem | 21:02 |
| sdake | various setups | 21:02 |
| sdake | without having to wake up at 4am to do the testing | 21:02 |
| rhallisey | once I have it cached it goes a little quicker | 21:02 |
| sdake | after sam has fixed theproblem | 21:02 |
| sdake | yup | 21:02 |
| sdake | but ya, its busted | 21:03 |
| sdake | i only printed out dhcp output | 21:03 |
| sdake | the othe services also seem to fail for various reasons | 21:03 |
| sdake | (neutron-agents) | 21:03 |
| rhallisey | I think I know how.. | 21:05 |
| rhallisey | or why rather | 21:05 |
| rhallisey | they keep running over eachothers config | 21:05 |
| rhallisey | need to run set_configs.py in supervisord | 21:05 |
| rhallisey | or in the first service run | 21:05 |
| *** akwasnie has quit IRC | 21:09 | |
| rhallisey | stuck on haproxy.. | 21:10 |
| rhallisey | crap I need to re build everything | 21:14 |
| rhallisey | this is going to take a while.. | 21:14 |
| *** daneyon_ has joined #kolla | 21:16 | |
| *** daneyon has quit IRC | 21:20 | |
| * rhallisey builds for next 2 hours | 21:22 | |
| *** daneyon has joined #kolla | 21:23 | |
| sdake | building docker images is the best part of my job | 21:24 |
| sdake | buildbuild build - 30 seconds of debug | 21:24 |
| sdake | buidl build build for 15 inutes | 21:24 |
| sdake | 30 seconds of debug | 21:24 |
| sdake | rinse repeat | 21:24 |
| rhallisey | if you want to test I can push :) | 21:25 |
| *** sdake_ has joined #kolla | 21:25 | |
| *** daneyon_ has quit IRC | 21:26 | |
| *** sdake__ has joined #kolla | 21:27 | |
| *** sdake has quit IRC | 21:28 | |
| *** itsuugo has quit IRC | 21:29 | |
| *** sdake_ has quit IRC | 21:30 | |
| *** dwalsh has quit IRC | 21:31 | |
| *** dwalsh_ has quit IRC | 21:32 | |
| sdake__ | rhallisey I know what the issue is | 21:37 |
| sdake__ | set_configs is running 4 copies at once | 21:38 |
| sdake__ | it is not designed to run parallel | 21:38 |
| rhallisey | ya that's what I said | 21:38 |
| sdake__ | it is racing | 21:38 |
| rhallisey | I think you were dc'd | 21:38 |
| sdake__ | howdo we make it run just one copy? | 21:38 |
| rhallisey | I have the fix, but I have to rebuild all my images | 21:38 |
| sdake__ | paste teh diff | 21:38 |
| sdake__ | i'll test it | 21:38 |
| rhallisey | I changed the priority of dhcp | 21:39 |
| sdake__ | how would that help | 21:39 |
| rhallisey | and only ran set_configs.py in dhcp | 21:39 |
| sdake__ | the others would race | 21:39 |
| sdake__ | oh i see | 21:39 |
| sdake__ | does that mean dhcp runs unitl completion? | 21:39 |
| sdake__ | paste a diff, I'll build and test quickly | 21:39 |
| sdake__ | or pusha new review | 21:39 |
| sdake__ | push a new review, that willbe better | 21:39 |
| rhallisey | http://fpaste.org/272391/14434763/ | 21:40 |
| rhallisey | ok | 21:40 |
| rhallisey | I'm not sure if dhcp will run until completion | 21:40 |
| sdake__ | well then that wont work | 21:40 |
| sdake__ | (if it doesn't run until completion) | 21:41 |
| *** sdake__ is now known as sdake | 21:41 | |
| rhallisey | I have another strategy, but we won't use json stuff. The config-external stuff was never removed from neutron-agents | 21:41 |
| rhallisey | this fix would be like a 4 line change | 21:41 |
| sdake | strategy means something entirely different to you then it does to me :) | 21:41 |
| rhallisey | just don't source kolla-common | 21:42 |
| sdake | yes but the other agents would start up | 21:42 |
| sdake | without their files being copied | 21:42 |
| rhallisey | they will still be copied | 21:42 |
| rhallisey | cofig-external will be called | 21:42 |
| sdake | groan | 21:42 |
| rhallisey | look at what I removed in the patch | 21:42 |
| sdake | yes I dont think that will work | 21:43 |
| sdake | looks completely racy to me | 21:43 |
| rhallisey | neutron-agents had it's own built in config-extneral | 21:43 |
| rhallisey | ya but this use to work | 21:43 |
| rhallisey | the only change mad for neutron-agents is that kolla-common is overwriting CMD | 21:43 |
| rhallisey | a fix would be to remove sourcing it | 21:44 |
| rhallisey | and they would work again like they did before | 21:44 |
| rhallisey | with config-external | 21:44 |
| sdake | well the question is, did they work correclty with json copying? | 21:45 |
| sdake | they were in separate directories | 21:45 |
| sdake | the problem now is they are all in the same directory | 21:45 |
| rhallisey | I can just change it back | 21:45 |
| sdake | putting them all in one directory is what creates the race condition | 21:46 |
| sdake | why not just have 4 copies and 4 json files | 21:46 |
| rhallisey | that would require changing the python bits | 21:47 |
| rhallisey | for a 'fat' container that isn't going to be around forever | 21:47 |
| rhallisey | let me push a patch to show you what I mean | 21:48 |
| sdake | well i'm stumped i don't see a tidy solution | 21:49 |
| sdake | other then making set_config.py handle fat containers | 21:49 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Alternate solution to get Neutron Agents work with JSON refactor https://review.openstack.org/228638 | 21:50 |
| rhallisey | sdake ^ that will prevent $CMD from being run over when running kolla-common | 21:51 |
| rhallisey | since neutron-agents is fully self contained. It never used the base config-external | 21:52 |
| sdake | but it uses config-external.sh right? | 21:52 |
| sdake | which is not designed to read json blobs | 21:52 |
| rhallisey | ya a local config-external.sh | 21:52 |
| rhallisey | each of those services has a local copy | 21:52 |
| rhallisey | look in docker/neutron/neutron-agents/neutron-dhcp-agent | 21:53 |
| rhallisey | you'll see what I mean | 21:53 |
| rhallisey | since it never runs set_configs.py those errors you were seeing should go away and CMD should run as normal | 21:54 |
| sdake | yes but there is no json parsing of the file list? | 21:55 |
| rhallisey | and it will copy configs using the local cofig-external | 21:55 |
| rhallisey | so there's no json here | 21:55 |
| sdake | its got the hard coded file copy | 21:55 |
| rhallisey | correct | 21:55 |
| sdake | somethign just seems wrong about that | 21:55 |
| sdake | the idea is to get rid of config-external.sh | 21:56 |
| sdake | not have a special case of it | 21:56 |
| rhallisey | well it's how we use to do it. Besides that, because of the fat container, command will never get full use anyway | 21:56 |
| rhallisey | it will get hard coded in the start script | 21:56 |
| rhallisey | so I can implement fat container support into the python bits, but the problem is doing that without having to change everything | 21:58 |
| rhallisey | I intentionally left the config alone for neutron-agents because it was a fat container and a self contained unit with config-external | 21:58 |
| sdake | i suspect we will be stuck with fat containers forever because of neutron and docker upstream business prioirities | 21:58 |
| rhallisey | oh really.. | 21:59 |
| sdake | the feature is a 3 character change to get us to having non-fat container support | 21:59 |
| sdake | but it breaks docker machine | 21:59 |
| sdake | badly breaks | 21:59 |
| sdake | from waht I can tell there is no way to make them work together the way we need | 21:59 |
| sdake | basically, docker thinks its more important for machine to work then to have the feature we need to make thin containers work for neutron | 22:00 |
| sdake | which makes sense from docker's perspetive | 22:00 |
| sdake | i am deploying your test patch now | 22:00 |
| sdake | to see if it works | 22:00 |
| rhallisey | kk | 22:00 |
| sdake | i dont like how we have two interfaces into the container | 22:00 |
| sdake | i'd prefer to change the python bits | 22:01 |
| sdake | and have one interface | 22:01 |
| rhallisey | ok | 22:01 |
| *** mat425 has quit IRC | 22:01 | |
| sdake | even if that means the interface changes | 22:01 |
| sdake | the time to chang ethe interface is now | 22:01 |
| sdake | before we pull the trigger on 1.0 | 22:01 |
| sdake | boy i wish yo uwould have brought this up earlier | 22:02 |
| sdake | rather then telling me on release day :( | 22:02 |
| rhallisey | ya sorry. I didn't know how this would've affected us | 22:02 |
| sdake | yes, that is called risk, which means communication is necessary :) | 22:03 |
| rhallisey | I thought we were going to thin neutron at some point | 22:03 |
| sdake | we will if we ever can | 22:03 |
| sdake | but that doesn't mean we carry two interfaces into the container runtime | 22:03 |
| rhallisey | ya, but it sounds like almost never | 22:03 |
| sdake | that is the fundamental problem with that thinking | 22:03 |
| sdake | that we carry two interfaces with our 1.0 api release | 22:03 |
| sdake | we need one interface | 22:03 |
| sdake | the one interface is the json communication mechanism | 22:04 |
| sdake | i think a more complex solution which would work is to use a file lock | 22:05 |
| rhallisey | ugh this gets tricky.. | 22:07 |
| rhallisey | without know what the json files are named | 22:07 |
| sdake | https://docs.python.org/2/library/fcntl.html#fcntl.lockf | 22:07 |
| rhallisey | maybe if config.json isn't present I can just look for *.json | 22:07 |
| sdake | that is racy | 22:07 |
| rhallisey | ya it is | 22:08 |
| rhallisey | crap | 22:08 |
| sdake | fcntl.ockf would work perfectly | 22:08 |
| sdake | it is designed for this exact problem | 22:08 |
| rhallisey | ok then... I could have an env variable with the name | 22:08 |
| sdake | that is adding shit to the api | 22:08 |
| rhallisey | k looking | 22:08 |
| rhallisey | could it still be racy if we only lock in python though | 22:10 |
| sdake | as long as there is no deletion | 22:12 |
| sdake | there is currently deletion | 22:12 |
| sdake | i dont understand why you can't put them in separate directories as was with the original work | 22:12 |
| rhallisey | will this lock prevent 2 python scripts from running at the same time or lock the python script in until ocmpletion | 22:12 |
| sdake | right | 22:12 |
| sdake | but if you delete a file after its been loaded by another service | 22:13 |
| sdake | that would probablyreak something | 22:13 |
| sdake | i wonder if there is a coutning lock in fcntl | 22:14 |
| * sdake reads man page | 22:14 | |
| rhallisey | so I could put them in separate dirs, but sam wanted them all in one dir | 22:14 |
| rhallisey | though I don't know if that would help | 22:14 |
| rhallisey | unless you talking about individual json files | 22:14 |
| sdake | separte dirs would solve the deletion problem | 22:14 |
| sdake | we can't delete | 22:14 |
| sdake | i dont understand why we delete presently | 22:14 |
| sdake | do you knwo why? | 22:15 |
| rhallisey | sam added that. That would do a replace of an existing config | 22:15 |
| rhallisey | actually why would you need that.. | 22:15 |
| rhallisey | wouldn't you just start a new container if changing your json ?>> | 22:15 |
| rhallisey | I think you would have to | 22:16 |
| rhallisey | and why can't we just copy over instead of delete then copy | 22:16 |
| * rhallisey looks | 22:16 | |
| *** dims has quit IRC | 22:17 | |
| rhallisey | I suppose we can remove the deletion. There shouldn't ever be anything to delete | 22:18 |
| sdake | why is the deletion there? | 22:21 |
| rhallisey | I'm not sure. I can't really think of a use case | 22:22 |
| sdake | ya copy over should be fine | 22:22 |
| sdake | if we have copy over we can use fcntl to lock | 22:22 |
| sdake | its a little bit more complicated then not using fcntl,but posix locks have been around since 90s | 22:23 |
| sdake | or 80s maybe | 22:23 |
| sdake | i really wish I understood why sam has that delete in there | 22:23 |
| rhallisey | how about instead of copy over just copy once | 22:23 |
| rhallisey | if it exists just leave it alone | 22:23 |
| sdake | that doens't work for the copy_always mode | 22:24 |
| rhallisey | ok true | 22:24 |
| sdake | copy always could be problematic as well | 22:25 |
| sdake | we could partially copy a file while a different service has loaded it | 22:25 |
| sdake | after the lock was released | 22:25 |
| sdake | which would result is really hard to solve partial data bugs | 22:25 |
| sdake | what we need is to implement a counting semaphore with fcntl | 22:26 |
| sdake | I think that wouldsolve the problem nicely | 22:26 |
| rhallisey | yikes. Are we sure supervisord won't hep us here at all | 22:26 |
| sdake | no idea | 22:26 |
| sdake | it may have somethig helpful | 22:27 |
| sdake | RT FM | 22:27 |
| rhallisey | I feel like it would | 22:27 |
| sdake | check out the manual | 22:27 |
| sdake | i would assume it has dependencies | 22:28 |
| sdake | fi we can just run the copy operation one time | 22:28 |
| sdake | that would solve the problem | 22:28 |
| sdake | copy runs (set_config) as one service | 22:29 |
| sdake | other services depend on it | 22:29 |
| sdake | and dont do set_config | 22:29 |
| sdake | or source kolla_common.sh or whatever the current magic is | 22:29 |
| rhallisey | https://github.com/Supervisor/supervisor/issues/122 | 22:29 |
| rhallisey | so we could run one of the services by hand first | 22:31 |
| rhallisey | or | 22:31 |
| rhallisey | let's just run the python script | 22:32 |
| rhallisey | before supervisod in the dockerfile | 22:32 |
| sdake | you can't run two scripts like that back to back | 22:32 |
| sdake | there is only one entrypoint | 22:32 |
| sdake | you could make a scirpt which calls one then the other however | 22:32 |
| sdake | which is pretty terribad :) | 22:32 |
| rhallisey | RUN python set_configs.py :P why not | 22:33 |
| sdake | RUN runs at build time not runtime | 22:33 |
| rhallisey | don't use it as an entrypoint | 22:33 |
| rhallisey | err | 22:33 |
| rhallisey | that's true | 22:33 |
| rhallisey | ya haha | 22:33 |
| sdake | btw, i tested your patch it does not work | 22:33 |
| sdake | the second one | 22:33 |
| rhallisey | run supervisordd as a script | 22:33 |
| rhallisey | oh really :( | 22:34 |
| sdake | see if there are dependencies in supervisord | 22:34 |
| sdake | all the config files end up in neutron-agent | 22:34 |
| sdake | and I think they need to be in other dirs as well but not sure | 22:34 |
| sdake | if supervisord has dependencies we can justuse that | 22:34 |
| rhallisey | startsecs=5 | 22:34 |
| sdake | not really liking that much | 22:34 |
| sdake | i really hate sleep hacks | 22:35 |
| rhallisey | I don't either | 22:35 |
| sdake | noob programming 101 | 22:35 |
| sdake | dependencies | 22:35 |
| sdake | supervisord must have dependencies | 22:35 |
| rhallisey | 6 days ago | 22:35 |
| rhallisey | +1 ?? This is from May, 2012, I think that supervisor will never have dependencies. | 22:35 |
| rhallisey | Someone could suggest alternatives with dependencies management? | 22:35 |
| rhallisey | :( | 22:35 |
| rhallisey | i'll keep looking | 22:35 |
| rhallisey | looks like it's sleep or a script | 22:35 |
| sdake | http://stackoverflow.com/questions/17443692/python-supervisord-program-dependency | 22:36 |
| sdake | the last example looks good | 22:37 |
| rhallisey | hehe nice | 22:38 |
| rhallisey | ok. I'll add that into the first patch | 22:38 |
| sdake | ok i think the shell script on entry with starting set_config followed by supervisrod is the best way to go | 22:38 |
| sdake | so basically a meta start.sh | 22:39 |
| sdake | that will always work correctly | 22:39 |
| sdake | or we can use the last example and wait for set_configs to exit | 22:41 |
| rhallisey | ok so this will wait for dhcp to start or just run set_configs | 22:41 |
| sdake | two operations, run set config, then run supervisord | 22:41 |
| sdake | i'm not sure if the process reaping will work correctly with that or not | 22:41 |
| sdake | but our processing reaping in kolla is fubared anyway | 22:42 |
| sdake | fkn docker | 22:42 |
| sdake | let me look at the supervisord code a bit sec | 22:42 |
| sdake | what does startsec do? | 22:44 |
| sdake | dely for # of seconds before starting? | 22:44 |
| rhallisey | ya | 22:47 |
| *** banix has quit IRC | 22:50 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Neutron Agents json support https://review.openstack.org/228611 | 22:52 |
| sdake | have a look at that | 22:52 |
| sdake | we dont need a delay in there either | 22:52 |
| sdake | jsut slows things down with no gain | 22:52 |
| rhallisey | you beat me to it | 22:53 |
| rhallisey | was about to push | 22:53 |
| rhallisey | mine has a loop, but I don't see why it's needed | 22:53 |
| rhallisey | I just make sure $CMD exists | 22:53 |
| sdake | getting ready to deploy | 22:54 |
| rhallisey | kk | 22:54 |
| rhallisey | that should work | 22:54 |
| rhallisey | assuming the json stuff I added is correct there | 22:54 |
| rhallisey | I think it is | 22:54 |
| sdake | rhallisey remove dyour workflwo vote | 23:00 |
| rhallisey | it's only -1 on the alternate solution | 23:01 |
| rhallisey | refresh | 23:01 |
| rhallisey | sdake, did that work for you? | 23:03 |
| sdake | still deploying | 23:03 |
| sdake | my 750 doesnt'speed up deploy times | 23:03 |
| sdake | since i deploy to a different machine | 23:03 |
| sdake | and the different machine is runningwith overlayfs | 23:03 |
| sdake | which is already much faster then btrfs | 23:03 |
| sdake | ERROR:__main__:The source to copy does not exist: /opt/kolla/config_files/fwaas_driver.ini | 23:06 |
| rhallisey | ls /etc/kolla/neutron-agents | 23:07 |
| rhallisey | is fwaas_driver.ini in there? | 23:07 |
| sdake | no, a directory called "neutron-agents" is in that directory ;) | 23:08 |
| rhallisey | oh | 23:08 |
| rhallisey | what's in the next neutron-agents :) | 23:08 |
| rhallisey | do you find all the config files | 23:08 |
| sdake | chill young padowan | 23:08 |
| sdake | nothign is in opt/kolla from what i can tell | 23:10 |
| sdake | /opt/kolla that is | 23:10 |
| sdake | the j2 copies fwaas_driverini directly to /etc/neutron | 23:12 |
| sdake | rather then /opt/kolla/whatever | 23:12 |
| sdake | what directory should it be coming from | 23:12 |
| rhallisey | /opt/kolla/config_files | 23:12 |
| rhallisey | I'm confused, I mounted /etc/kolla/neutron-agents to /opt/kolla/config_files | 23:13 |
| rhallisey | can you rebuild --no-cache | 23:13 |
| sdake | look at the neutronagents dockerfie.j2 file | 23:13 |
| sdake | why would i rebuild --no-cache | 23:13 |
| sdake | the j2 file is wrong | 23:14 |
| sdake | reublidng --no-cache owuld take 5-10 minutes - no reason for that | 23:14 |
| rhallisey | oh I see it | 23:15 |
| rhallisey | well | 23:15 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Neutron Agents json support https://review.openstack.org/228611 | 23:15 |
| sdake | deploying now | 23:15 |
| rhallisey | nice | 23:16 |
| rhallisey | I was wondering where that was coming from. I thought you had bad docker caching or something | 23:17 |
| rhallisey | that issue drove me crazy for horizon | 23:17 |
| sdake | which issue | 23:17 |
| sdake | i have neverseen teh cache misbhehae really | 23:17 |
| sdake | but when I do something that i athink has cache consequences is I generally run with --no-cache ;) | 23:17 |
| rhallisey | I've had a few | 23:18 |
| sdake | did yo u finish taht review for jeff | 23:19 |
| rhallisey | it would drive me crazy. I'd build for 30min then run and get the same errors | 23:19 |
| rhallisey | I haven't | 23:19 |
| rhallisey | I will tonight | 23:19 |
| sdake | ok | 23:19 |
| rhallisey | I need to eat dinner and stuff | 23:19 |
| sdake | right | 23:19 |
| sdake | 4:20 | 23:20 |
| sdake | you know whta that means | 23:20 |
| bmace | you will be having the munchies soon? | 23:20 |
| sdake | haha I wish | 23:21 |
| sdake | obama take my kids away | 23:21 |
| rhallisey | lol | 23:21 |
| sdake | 9 minutes 42 seconds to deploy single node | 23:25 |
| sdake | did not work | 23:25 |
| sdake | trying small change | 23:27 |
| sdake | and also running with --no-cache | 23:27 |
| sdake | i dont get why perl modules re installed int he bae image | 23:29 |
| sdake | perl is not part of the system :( | 23:30 |
| sdake | fawk openstack-base is breaking on oslo.log | 23:31 |
| *** loth1 has joined #kolla | 23:32 | |
| *** loth has quit IRC | 23:33 | |
| rhallisey | sdake, hows it look | 23:50 |
| rhallisey | getting past neutron? | 23:50 |
| sdake | nah got jammed up on this patch not merging correclty | 23:50 |
| sdake | sorting it out now | 23:50 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Neutron Agents json support https://review.openstack.org/228611 | 23:56 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix oslo_log https://review.openstack.org/228669 | 23:56 |
| *** achanda has quit IRC | 23:58 | |
| sdake | building should hae reulsts in t-20 minutes | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!