Monday, 2015-06-08

*** vinkman has quit IRC00:03
*** sdake_ has joined #kolla00:06
*** sdake has quit IRC00:10
openstackgerritSteven Dake proposed stackforge/kolla: Make swift bashate compliant  https://review.openstack.org/18916600:10
openstackgerritMerged stackforge/kolla: Autogenerated value for DESIGNATE_POOLMAN_POOLID  https://review.openstack.org/18904300:18
*** sdake has joined #kolla00:32
*** sdake_ has quit IRC00:35
*** zhiwei has joined #kolla00:39
*** sdake_ has joined #kolla01:06
*** dims__ has quit IRC01:06
*** sdake__ has joined #kolla01:09
*** sdake has quit IRC01:09
*** sdake__ is now known as sdake01:09
*** sdake_ has quit IRC01:13
*** jtriley has joined #kolla01:26
*** jpeeler has joined #kolla01:26
*** jtriley has quit IRC01:31
*** erkules_ has joined #kolla01:32
*** erkules has quit IRC01:35
zhiweisdake: If you are using VIM, you can add these lines to ~/.vimrc file. http://paste.openstack.org/show/272544/01:40
sdakethanks zhiwei I am ware of that01:41
sdakebut it makes my screen bleed when I'm typing :)01:41
zhiweihaha01:41
zhiweiyou can remove the highlight line01:41
zhiweionly add the autoremove section.01:41
sdakei like my vi the  way it is :)01:42
sdake*default* :)01:42
zhiweiya01:42
sdakei'll submit another review if a review is busted ;)01:42
zhiweiok01:42
*** sdake_ has joined #kolla01:43
zhiweican we use ansible for both aio and multi node deployment?01:43
*** sdake has quit IRC01:46
sdake_i dont see why not01:47
sdake_just a matter of the inventory file01:47
openstackgerritSteven Dake proposed stackforge/kolla: Ansible multi-node specification  https://review.openstack.org/18915701:49
sdake_zhiwei in the future if you wouldn't mind voting +1 or -1, so I know the problems you found are met, i'd super appreciate it :)01:50
zhiweiok, will do that.01:51
sdake_ta01:51
*** sdake has joined #kolla01:54
*** sdake_ has quit IRC01:58
*** dims_ has joined #kolla01:58
*** dims_ has quit IRC02:03
sdakezhiwei the defeciency has been corrected, can you revote plz02:08
zhiweisdake: new comments added.02:23
*** sdake_ has joined #kolla02:25
*** sdake has quit IRC02:29
*** bmace has quit IRC02:31
openstackgerritSteven Dake proposed stackforge/kolla: Ansible multi-node specification  https://review.openstack.org/18915702:32
sdake_zhiwei ok problems should be fixed ;-)02:43
zhiweiI saw02:43
*** bmace has joined #kolla02:44
*** sdake has joined #kolla02:44
*** sdake_ has quit IRC02:48
*** dims_ has joined #kolla03:03
sdakebmace there?03:04
*** dims_ has quit IRC03:09
*** jtriley has joined #kolla03:15
SamYaplesdake: i am here03:16
sdakesamyaple for your viewing pleasure03:17
sdakehttps://review.openstack.org/18915703:17
SamYapleyup skimmed it03:17
* sdake has to work with someone from every timezone03:17
* sdake groans03:17
SamYaplegood work03:17
SamYapledo be fair, im only like 2 hours off of your timezone...03:17
sdakeKevin has some interesting thoughts in review #203:17
SamYapleto*03:17
sdakeI wouldn't do it if I didn't choose to ;)03:18
sdake(the tz thing)03:18
*** jtriley has quit IRC03:20
SamYaple" If the reason is to have containers not change once they start, you could pass the config files through a unique dir per container /var/lib/kola/onfdir/<id>/ or something. Then pass it through from the host."03:20
SamYaplehow ouwld that work?03:20
sdakebindmount /etc/blerg03:20
sdakecopy /etc/blerg/keystone.conf /etc/keystone.conf ; rm -f /etc/blerg/keystone.conf03:20
SamYaplebut the bndmount never goes away03:21
SamYaplethat is by far the cleanest solution and I like that idea for sure03:21
SamYaplebut the bindmount persists03:21
sdakeya I'm not quite sure what to make of it03:22
SamYapleinterestingly, through the docker api we could remove the bindmount without regenerating a new container03:22
SamYapleit would remove the ability to docker inspect and backup the container, configs and all03:23
SamYapleits dancing o nthe edge of non-immutable containers03:23
sdakeI agree03:24
sdakei have a preference for serialization/deserialization03:25
sdakesamyaple if there is anything I got wrong in the spec, please comment and vote on it so it can be corrected or merged ;-)03:27
*** dasm has quit IRC03:27
*** shadower has quit IRC03:27
*** bradjones|away has quit IRC03:27
*** dasm has joined #kolla03:28
SamYapleserialization/deserialization does work very weel, but it can be a bit confusing/opaque I agree03:28
SamYapleit can be used by anyone though, not just ansible03:28
*** bradjones has joined #kolla03:28
*** bradjones has joined #kolla03:28
sdakefeel free to comment with kevin's review as well03:29
sdakethere was a Q re security hidden in there ;)03:29
SamYapleanswered already03:29
SamYapleor replied too03:29
SamYaplethe inventory is _not_ optional03:29
sdakeif you reply to his comments, you have to click "review" on the review where you commented03:30
sdakehe was talking abou the unified config file to be optional03:30
sdakenot the inventory file03:30
sdakeI may just remove that language03:30
SamYaplei think kevin means to say the inventory should be optional, it is not03:31
sdakedepends what happens next week03:31
sdakerecommend asking in the review - the seciton talks about a unified config file03:31
SamYaplenot that section03:31
sdakeok03:31
sdakemaybe i'm too tired to function :)03:31
SamYapleits cool03:31
*** dasm has quit IRC04:54
*** jtriley has joined #kolla05:04
*** jtriley has quit IRC05:08
*** nihilifer_ has joined #kolla05:09
*** dasm|afk has quit IRC06:04
*** dasm has joined #kolla06:12
dasmgood morning :)06:12
SamYaplegood morning dasm06:20
sdakehey dasm06:26
sdakedasm highest priority review https://review.openstack.org/18915706:26
sdakeplz take a look ;)06:26
*** tobe has joined #kolla06:45
*** jtriley has joined #kolla06:52
*** jtriley has quit IRC06:58
*** inc0 has joined #kolla07:01
inc0good morning07:01
SamYaplehello inc007:01
SamYaplehow are you07:01
inc0we had long weekend in PL, so a bit lazy afterwards07:02
SamYaplePL?07:03
inc0Poland07:03
SamYapleoh cool07:03
inc04 days is enough to forget about work;)07:03
SamYapletell that to sdake07:04
inc0today is good day to resolve corporate paperwork and maybe just address few comments07:05
inc0or make nitpicks reviews;P07:05
inc0;)07:05
inc0speaking of which...Steven was busy07:06
inc0I've just seen gerrit07:07
SamYaplethe dake sleeps for no man07:07
*** fabiand has joined #kolla07:08
fabiandHey07:08
SamYaplehello07:09
inc0morning07:09
fabiandOh - this time zone also seems to be represented .)07:10
fabiand:)07:10
fabiandHey, I'm looking at the kolla dockerfiles, and wondered a bit about cinder.07:10
fabiandIsn't cinder also using an iscsi initiator?07:10
fabiandOr is it rather a target, and nova is using the initiator to connect to it?07:11
inc0well, I think it's nova who does connections07:11
inc0and cinder just exposes target, but I'm not sure about that07:11
fabiandMy reason for asking is around the question if someone ever tried to connect to iscsi targets from within the container07:12
fabiandinc0, right - that sounds reasonable to me ..07:12
fabiandWe also tried to run an iscsi initiator inside a container and ran into some trouble07:13
inc0I don't think we contenerize qemu, but we do contenerize nova-compute07:14
SamYapleinc0: we containerize everything07:14
fabiandinc0, so qemu is directly accessing the iscsi tgt by it's built-in iscsi support?07:15
fabiandwe do it indirectly by pointing qemu to a local device mapping the the remote target (so using iscsiadm to discover and attach the remote tgt)07:15
inc0I supose, but I never really looked at that07:15
fabiandRight, I'll loook at it as well.07:16
SamYaplehowever nova does it fabiand. ill check07:16
fabiandAnother issue we had is that - we meess with lvm etc - that lvm would not work correctly without udev07:16
SamYaplefabiand: it will actually work, but you have to mount in /dev07:16
fabiandAnd I wondered of cinder is also manipulating storage devices, or just exposing already available devices ..07:17
SamYaplethen the host udev can handle everything07:17
fabiandSamYaple, ah ..07:17
fabiandYou seee parts if you do that, you see the kernel events, but you don't get all the udev event's from the udev daemin07:17
fabianddaemon07:17
*** athomas has joined #kolla07:18
SamYaplefabiand: looks like iscsi is attached to the host, not using qemu initiator07:18
fabiandSamYaple, cool - thanks for looking07:18
fabiandThen I'd really be interested if that was tried, because here as well, udev played a role ..07:18
SamYapleusing the qemu initiator would actually resolve alot of the cinder headaches07:18
fabiand:)07:18
SamYaplehttp://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/qemu-built-in-iscsi-initiator.html07:19
inc0https://github.com/openstack/nova/blob/master/nova/virt/volumeutils.py#L2307:19
inc0fabiand, and btw, person you'd really want to talk to is rhallisey07:20
SamYapleyea fabiand he did all the cinder work07:20
SamYapleive only done cinder+ceph stuff. i abandonded lvm+iscsi long ago07:20
fabiandWhy did you abandon it, SamYaple ?07:20
fabiandDid someone take it over, or did you run into difficulties?07:21
SamYaplefor the reasons we are having the conversation xD07:21
fabiandinc0, SamYaple -- I'll then wait for him ..07:21
fabiandYou mean, it did nto work?07:21
inc0iscsi is hard to contenerize07:22
inc0and really not too many people use it on prod07:22
fabiand:)07:22
inc0ususally prod is ceph based07:22
fabiandRight07:22
SamYaplei didnt dig into it to much. ceph is a much better storage system for openstack in my opinion07:22
* fabiand comes from the oVirt land ... and there people do use iSCSI in production07:22
SamYapleits more openstack-ihs too07:22
fabiandYep, I think ceph works nicely in containers ..07:22
SamYaplefabiand: dont feel bad, its all iscsi where I work07:23
fabianda lot of the device stuff is not there.07:23
inc0and it's *way* more scallable07:23
fabiandI believe qemu access ceph directly?07:23
SamYapleyes07:23
fabiandThere we goo ..07:23
fabiandThen you don't have these core issue with udev07:23
fabiandudev really is a bummer in the container land07:23
* SamYaple shudders07:23
SamYapleudev ugh07:23
fabiandSamYaple, I don't feel bad about iSCSI, just getting it containerized07:23
inc0nope, ceph does all that and it exposes block devices07:23
fabiandinc0, right ...07:24
inc0nova can use it for ephemeral disks as well07:24
inc0out of the box07:24
SamYapleinc0: kinda. ceph in containers is still a bit annoying _if_ you let container mount the disk07:24
fabiandOh well .. okay, it exposes block devices, and qemu is using them ..07:24
fabiandSamYaple, would you elaborate on that? I mean - who should mount the disk, if not the container?07:24
SamYapleI mount it external with the host and bindmount it into the container07:25
fabiandAwh! That's cheating :)07:25
SamYapleotherwise youll have issue with the mount namespace disappering during a container restart07:25
fabiandYes, I see ..07:25
SamYaplenah its the only safe way t ogo!07:25
fabiandWell, you could have a data container :)07:26
SamYaplei care about data integrity07:26
SamYaplenot with ceph, since youll only get performance when its talking to the raw disk for journaling07:26
inc0don't forget we don't care about separation *that* much - kolla isn't supposed to be multitenant project07:26
fabiandRight ..07:26
SamYapleinc0: what do you mean by that?07:27
SamYaplei most certainly care about seperation07:27
fabiandinc0, I was about to throw that in - isolation, you give up isolation ...07:27
fabiandBut yes, I - I agree, sometimes you don#t care about that aspect of containers.07:27
*** jasonsb has quit IRC07:27
inc0I mean if container plays with base system, when it has to (kernel, storage), we can run priviledged containers07:27
fabiand(limited)07:27
SamYapleyou guys do know oyu can still access all the container storage in /var/lib/docker right?07:27
*** jasonsb has joined #kolla07:28
inc0what I mean is we don't do that lightly, but we can do that07:28
inc0otherwise things would be much uglier07:29
* fabiand sees that cinder is manipulating LVM .. so I'm curious to speak with rhallisey07:29
SamYapleagreed. im still not happy about everything running as root in the container. but i havent got around to patching in all the yaodu stuff to fix that yet07:29
SamYapleroot in container == root on host07:29
fabiandYes ..07:29
fabiandYou also neet to be in peermissive mode with that many bind mounts form the host side, right?07:30
inc0I'm pretty sure there will be things which won't be doable without priviledged mode07:30
SamYapleinc0: things will be harder to do, but root is root07:30
inc0still, since kolla is deployment method, I don't think this is that much of an issue07:30
inc0I mean, people would deploy this stuff as root anyway07:31
SamYapleno its pretty important. its just a few lines and we can have a MUCH more secure project.07:31
fabiandI also saw that some (maybe neutron?) are using --net=host ... How is the experience with that so far.07:31
fabiandAny performance impact or management difficulties?07:31
SamYaplefabiand: everything uses --net=host07:32
SamYapleits for performance07:32
fabiandOh, everything ..07:32
fabiandRight07:32
SamYapledocker-proxy is slow as dirt07:32
SamYaplesomething like a ~20ms delay per packet07:32
fabiandYeah, I read that in the spec07:32
*** jasonsb has quit IRC07:32
SamYaplepersonally i like --net=host, but it requires a bit more tying container to host in the config files07:33
fabiandYes ..07:33
fabiandI thought about an approach where the container claims all the devices which are not yet configured when it get#s started ..07:34
SamYapleonly a bit though. some stuff would still need those per host configs (mariadb/galera, rabbitmq, some openstakc services)07:34
fabiandThen you can leave like one device to the host, for mgmt/communication and the rest is moved into the container ns ..07:34
SamYaple:) i like ideas like that. i would like to see an implementation07:35
fabiandSamYaple, http://dummdida.tumblr.com/post/118686497295/moving-unconnected-nics-into-a-container-namespace -- just needs to be pulled into the container07:35
SamYapleah i see. this isn't something the container automatically does07:36
*** mstachow has joined #kolla07:36
*** mstachow has quit IRC07:37
harmwsdake: sup07:50
*** gfidente has joined #kolla07:51
harmwsdake: if its about https://review.openstack.org/#/c/189157/4/specs/ansible-multi.rst, Ill be heading down ansible-road from this week on - now that designate got merged, finally :)07:55
*** dims_ has joined #kolla08:28
*** dims_ has quit IRC08:33
*** shadower has joined #kolla08:34
*** jtriley has joined #kolla08:41
*** jtriley has quit IRC08:46
*** weiyu-001 has joined #kolla08:52
*** inc0 has quit IRC08:57
*** inc0 has joined #kolla08:59
openstackgerritPaul Bourke proposed stackforge/kolla: Documentation improvements and minor fixups  https://review.openstack.org/18874208:59
*** jasonsb has joined #kolla09:02
fabiandmh, btw - SamYaple inc0 -- did you experience selinux issues when biond mounting much stuff from the host?09:04
fabiandI imagine that you'll need to turn off selinux to get freedom when you bind mount in a lot of paths from the host.09:05
fabiandBut I haven't investigated that area so far ..09:05
SamYaplewe make turning selinux off a requiremnt for kolla09:05
fabiandhehe09:08
fabiandWill it stay like that?09:08
fabiandOr do you plan to somehow get along with selinux?09:09
SamYapleprobably09:09
SamYapleits more a docker+selinux thing09:09
SamYapleit has its own issues09:09
fabiandYeah ..09:09
SamYaplei dont think anyone is against supporting it, but it isnt possible at the moment09:10
fabiandRight09:10
SamYaplei honestly havent dug into it for quite a while09:10
fabiandYeah ..09:10
fabiandIt will be quite a mess IMO09:10
SamYaplei agree09:10
fabiandI.e. I'd imagine that you get denials for each bind mount you use ..09:10
fabiandetc09:11
*** jasonsb has quit IRC09:13
SamYapleit also requires kolla ot maintain profiles for selinux and inject them on the host09:13
SamYaplei would like to get there htoug09:14
fabiandYeah, sure, that would be a solution09:14
*** inc0 has quit IRC09:27
*** inc0 has joined #kolla09:33
*** zhiwei has quit IRC09:55
*** sdake has quit IRC10:00
*** inc0 has quit IRC10:01
*** openstackgerrit has quit IRC10:09
*** dims_ has joined #kolla10:09
*** openstackgerrit has joined #kolla10:09
*** shardy_ has joined #kolla10:19
*** shardy has quit IRC10:21
SamYaplepdb: you around?10:24
pdbSamYaple: howdy10:24
SamYaplehey!10:24
*** shardy_ has quit IRC10:25
SamYapleper your ansible spec review10:25
SamYaplecan you define what ansible generated configs into a container would look like?10:25
*** shardy has joined #kolla10:25
pdbwell, originally in an implementation I toyed with internally, we had ansible push configs out to target nodes, then bind mounted them to containers. However given bind mounts are something we want to avoid, I wondering if there's a way we can store them in data vols instead10:27
SamYapleit would require a recreation of the data vol to update the configs10:27
SamYaplebindmounting is what i did in yaodu as well10:27
SamYapleit worked fine, except you had permissions issues10:28
pdbI think that's the main reason we used bind mounts10:28
pdbpermissions should be ok so long as you store the configs somewhere the ansible user can write to10:29
SamYaplepdb: nope. doesnt work like that10:29
SamYapleyou have to match the uid from the user in the container to the file10:29
SamYaplewith yaodu I solved this with a "yaodu" user and group on the host, not ideal for sure10:30
*** jtriley has joined #kolla10:30
SamYapleI am open to other suggestions. I have been thinking about this issue for ~4 months now. The generated configs are the cleanest solution I can find with the least amount of maintenance10:31
SamYaplethe env variable to config option mapping of crudini is not sustatainable in my opinion10:31
pdbyeah I would not be in favor of maintaing the crudini option at all tbh10:31
SamYaplewell that has to stick around to support triple-o downstream (unless that can be eventually changed)10:32
SamYaplebut it doesnt have to be the default/required10:32
pdbIm probably lacking context on this, but it feels we're making this more difficult for ourselves just to support a project that only potentially might use kolla10:34
pdbs/this/things10:34
*** jtriley has quit IRC10:35
SamYapleI understand. sdake is leading this and his stance is we dont leave people/projects behind. I agree with him, but it is on those who use crudini to maintain it10:35
SamYapleim more interested in your thoughts on the dynamic config since that is the blocker for moving forward10:35
pdbyeah I'll have a think about it some more and let you know10:37
pdbsort of curious about this permissions with bind mounts you mentioned as I really dont recall having issues there10:38
SamYapleyou were running as root in the container10:38
pdbyes :)10:38
SamYaplethat is why it worked10:38
pdbthought so10:38
SamYapleand why its bad10:38
SamYaplein yaodu i did it proper with user and dropping privs10:39
SamYaplewell do that in kolla too, but only so many hours in the day10:39
pdbthat's fair enough. From what Ive heard bind mounts aren't an option anyway?10:39
SamYapleeveryhting is an option, but sdake makes valid point against them that I cant argue10:39
SamYaplepdb: your comment about other non configini configs is valid10:42
SamYaplei had thought about this10:42
SamYaplethere is an ugly option, that works well10:42
SamYaplebase64 the non config-ini configs and convert them in much the same manner10:42
pdbhow does that work with the augment files though?10:43
SamYapleso fun fact about the augment files. the merging happens outside the container10:44
SamYapleit is just an ansible module i wrote to merge config-ini files (can be run as native python file with some tweaking)10:45
SamYaplethe rabbit configs are wierd because erlang is wierd10:45
pdbyeah what I was going to say10:45
SamYaplebut could be merged just the same10:45
SamYaplenot the same module, but a new one for rabbit10:45
pdbregardless of where it happens, you'll have to end up writing custom merge code for each config format?10:46
SamYaplemysql is ocnfig ini parsible10:46
SamYaplehow many custom configs are there :)10:46
SamYaplerabbit and configini10:46
SamYaplebetween those two thats all of openstack10:46
pdbhaproxy10:46
SamYaplethats a hole nother thing10:47
SamYaplewhole* other*10:47
SamYapleso that one is very tricky, but has nothing to do with the generated/augment stuff10:47
SamYapleand it depends on what we want to allow being configurrable10:47
SamYapleI believ ethe fully configurable bit was only for openstack configs (though I would like to extend it as far as feasible)10:48
pdbI need to revisit sdake's arguments around bind mounts. I think you're approach is a decent compromise but do see it as being complex and there will be complications in the serialisation/deserialisation rigmarole10:52
pdbs/you're/your/10:52
SamYapleyea i agree. i would love to make it less ocmplicated10:52
SamYaplebind mounts were way cleaner imo10:52
pdbits much easier for the user and newcomers to understand, that probably shouldn't be underestimated10:53
SamYapleplus it lets you keep the traditional 'from host modify /etc/keystone/keysotne.conf and restart container' approach10:53
pdbwhat I was about to say10:53
pdb:)10:53
SamYapledont get me wrong, it is very clean and works well10:53
SamYapleif the community is ok with a seperate kolla user it is also secure10:54
SamYaplei chose are random uid and guid, but they were static assignments10:54
SamYaple42424 i think10:54
pdbIt's not really related to my views but we have had customers here say they want the option of modifying configs on the target for debugging/testing/etc10:54
pdbI guess depending on the arguments for/against we could make it another deploy option10:55
SamYapleas long as I can do dynamic config generation and privelege dropping (running as a user, not root) im happy10:55
SamYaplebut keep in mind, its a config per container10:55
SamYaplewe would have nova-api.conf nova-compute.conf nova-scheulder.conf10:56
SamYaple(well nova-compute.conf was a different ocnfig in the past, but you get my point)10:56
pdbis that a problem?10:56
SamYapleno, but its not /etc/nova/nova.conf is my point10:57
pdbsure10:57
SamYaplerealisitcally it would be /opt/kolla/nova/scheduler.conf10:57
SamYapleor something like that10:57
SamYaplethat file would be /etc/nova/nova.conf in the container10:57
pdbyeah that's how I did it10:57
SamYaplesame on yaodu :)10:57
pdbseeing a pattern here ;)10:58
SamYapleKISS10:58
pdbyup10:58
SamYapledamn. now im going to have to have a conversation about this.10:59
* SamYaple sighs10:59
SamYaplei suppose we need a poll from the community on what they want. ie configs modifiable outside of docker and bindmounted in or truly idempotent containers11:00
pdbare bind mounts any less idempotent than the env file?11:02
SamYapleyea11:02
SamYaplethe env cant change in the container11:02
SamYapleif its bindmounted it can11:02
pdbyeah but realistically the container wont change once the service has started11:03
SamYapleidempotent is about restarting11:03
pdbwhat if I change the env file though and restart11:03
SamYaplewit hthe env method itll get overwritten11:04
SamYaplebindmount the change takes effect11:04
pdbare you not startng the container with something like --env-file=openstack.env11:04
SamYaplekinda? best way to put it. mostly no11:05
SamYaplehttps://gist.github.com/SamYaple/444e20b067695e8ac83811:05
SamYaplepdb ^11:05
*** tobe has quit IRC11:26
*** inc0 has joined #kolla11:51
*** jtriley has joined #kolla12:19
*** rhallisey has joined #kolla12:20
*** ChanServ sets mode: +o rhallisey12:23
*** rhallisey is now known as rhallisey|pto12:23
*** jtriley has quit IRC12:24
*** nihilifer_ has quit IRC12:28
*** shardy_ has joined #kolla12:36
*** shardy has quit IRC12:38
*** pdb has quit IRC12:41
*** pdb has joined #kolla12:42
*** shardy_ has quit IRC12:42
*** shardy has joined #kolla12:43
*** fabiand has quit IRC12:43
*** prad has joined #kolla12:45
*** fabiand has joined #kolla12:55
*** fabiand has quit IRC12:55
*** fabiand has joined #kolla12:55
*** fabiand has quit IRC12:59
*** fabiand has joined #kolla12:59
*** dwalsh has joined #kolla13:28
*** jtriley has joined #kolla13:30
*** jtriley has quit IRC13:35
*** jtriley has joined #kolla13:45
*** sdake has joined #kolla13:48
pdbSamYaple: what was the reasoning behind .buildconf rather than making it a parameter?13:52
*** gfidente has quit IRC13:52
sdakemorning please review https://review.openstack.org/#/c/189157/13:53
*** jtriley has quit IRC13:54
pdbsdake: hi, thanks for the reply. was discussing it somewhat with SamYaple earlier this morning (our morning :))13:57
sdakediscuss in review plz :)13:58
pdbok13:58
*** sdake_ has joined #kolla14:00
SamYaplepdb: i didnt do .buildconf but i am trying to get rid of it ;)14:03
pdbSamYaple: ok, that reminds me I still need to run my blueprint past sdake14:03
pdbSamYaple: I feel I would remove them in favor of a parameter to the build script14:03
pdbsdake: would you mind having a look at this if you get a chance? https://blueprints.launchpad.net/kolla/+spec/refactor-base-image-layout14:04
*** sdake has quit IRC14:04
SamYaplepdb: the issue with that is the differences in the scripts that may pop up with different distros14:05
SamYapleim ok with it, but this means added if;then;else blocks for every distro14:06
pdbSamYaple: yes, you had suggested that approach and I think it sounds good14:06
SamYapleoh then we have discussed this14:06
pdbI believe it's how devstack does it14:06
SamYaplelets not base what we do on devstack14:07
*** fabiand has quit IRC14:07
*** fabiand has joined #kolla14:08
*** jtriley has joined #kolla14:08
*** fabiand has quit IRC14:09
*** vinkman has joined #kolla14:10
openstackgerritMichal Jastrzebski (inc0) proposed stackforge/kolla: Keepalived container  https://review.openstack.org/18798114:16
sdake_inc0 did you have a chance to review the blueprint on ansible14:17
inc0sdake_, unfortunately I haven't I had cluster of meetings today14:18
inc0I'll do that first thing tomorrow14:18
sdake_groan14:18
sdake_i'll be out of the office tomorrow14:18
inc0ok, I'll look at it now then14:18
sdake_thanks :)14:18
sdake_love you long time ;)14:19
sdake_samyaple what are your thoughts on a data volume for config data vs serialization/deserialization14:19
sdake_i had this idea some time ago as well14:20
pdbsdake_: in spirit of discussing on the review my reply is there now14:20
SamYapledata volume will not work14:22
SamYapleit would have to be a bind mount14:22
sdake_pdb i put your blueprint in the discussion state and added some discussion to the whiteboard14:22
sdake_feel free to respond at your liesure :)14:22
pdbsdake_: thanks very much14:23
sdake_well it would be a -volumes-from which is a bindmount but we should be ok with volume-from bindmounts14:23
sdake_is that the type of bindmount you speak of or something else?14:23
SamYapleno, bind mount host files14:24
SamYapleusing a "data container" would not work14:24
SamYaplethere is no way t oupdate those files14:24
inc0sdake_, uhh...having 2 different configs might bite us someday14:27
sdake_delete the data container make a new one?14:27
vinkmanHmmm… I would vote against bind mounting host files into the containers… It opens the door to people doing all kinds of random things to the configurations…14:28
sdake_vinkman yes that idea is definately vetoed by everyone involved :)14:28
inc0vinkman, who "people"? admins?14:28
sdake_anyone with access14:28
vinkmananyone with access to the host...14:29
sdake_it violates immutability and the declartive nature of properties14:29
sdake_of containers that is14:29
inc0do we care about admins breaking their own cluster?14:30
pdbyou can lock down the permissions of the configs to the ansible user14:30
sdake_yes inc014:30
sdake_inc0 the installation should take precautions to protect the admin from themselves if possible :)14:31
sdake_if they rm -rf / then what can we do? :)14:31
sdake_but an admin might think its ok to go mucking with config variables in a live running container14:31
sdake_which is definately not ok14:32
sdake_samyaple do you mean it is technically not possible?14:32
sdake_sorry to be dene here, but i want to get to the root of the discussion with a data container14:32
pdbsdake_: I think some admins want it to be as easy as possible to tweak configs14:33
*** vinkman has quit IRC14:34
*** vinkman has joined #kolla14:34
pdbI understand you want to help prevent people shooting their own foot off but no point going out of the way to limit them either14:35
pdbrebuilding/redeploying containers for every config change is really cumbersome14:36
inc0especially if we consider things like neutron restarts14:36
inc0we'll be working on ways to reread configs without restarts, and forcing people to redeploy would be even worse14:37
SamYaplesdake_: yea if yo udelete the volume and recreate it everything has to be restarted14:37
SamYaplewhich is the same exact issue14:37
SamYaplecontainers are immutable14:37
SamYapleand for the record, I am not against it, i just went with the flow here. This was only brought back up because people didnt like the admittedly complicated encoded ENV vars14:38
SamYaplethe options I see are the ENV vars, or host bind mounts14:39
pdbor both ;)14:39
SamYapleyea I dont think so14:39
SamYaplei mean thats a really different decision14:39
SamYapleit affects the way the containers build too14:39
vinkmanYeah, I would agree mixing the two would be the worst of both worlds…14:41
SamYaplevinkman: technically anyone can do random things with the config anyway. they stil lexist on the host14:42
pdbwell at the end of the day the service just needs to be pointed at a config. how that config gets there can be variable14:42
SamYaplepdb: but it changes the deployment method for hte containers14:43
pdbtrue, and probably a lot of other things14:43
SamYapleno, not much else. But it would double the LOC for the container starting part14:44
inc0sdake_, quick question about ansible14:46
inc0why we'd even want to keep crudini when we'll have ansible?14:46
SamYaplesdake_ loves ansible questions!14:46
SamYapledownstream consumes crudini inc014:46
pdbI feel there's a lot of admins who will want more control over their configs. The bind mount option also opens up the possibility of using their own orchestration e.g. chef/puppet etc.14:47
SamYaplepdb: sdake_: datavolume for configs wit hthe option of bind mounting?14:47
inc0or rather, why not prepare config files outside container and push this to container on deploy?14:47
inc0SamYaple, pdb I mean, instead of making cruds inside container, predefine config file outside and just copy it in14:47
inc0then puppets and so on will have even easyier time14:47
inc0and our ansible stuff will just create this file14:48
SamYapleinc0: "predefine" can you expand?14:49
inc0so. we'll create /tmp/keystone.conf14:49
inc0for example14:49
inc0with env vars or ansible or puppet or vim or even emacks14:49
inc0emacs*14:49
inc0and all kolla will do is copy this little file to place we want it to be14:50
SamYapleinc0: how does it copy said file14:50
inc0COPY /said/file /etc/said/file?14:50
inc0in Dockerfile?14:50
SamYaplethats only at build14:50
SamYaplehow will people add what options they want14:50
SamYaplehow will you change the config14:51
inc0hmm...data volume for configs? ;)14:51
inc0there should be a way to push file into running container?14:52
SamYaplerecreating the data volume with each change is cumbersome as technically all files for all containre running on that host would be touched14:52
SamYapleinc0: you can, but thats the worst way to do it14:52
inc0maybe just VOLUME /tmp/kolla and in start.sh cp /tmp/kolla/keystone.conf /etc/keystone/14:52
inc0?14:52
SamYapleyou cant cp from outside the container to inside the container14:53
inc0or even VOLUME /etc/keystone ?14:53
SamYaplenone of this solvess the issue14:53
inc0mounting etc wouldn't?14:54
SamYaplemounting from where14:54
inc0host14:54
inc0host will hold sum of /etc/ of containers14:54
SamYaplethats what we are talking about. "bindmounting" thats what sdake said "those with access" are against14:54
SamYaplethat would be my prefered method honestly14:55
inc0mine too, admins are used to have configs in /etc14:56
inc0we'll use containers just as app runtime environment, not separation tool14:56
inc0multitenancy like separation I mean, we keep deps separation, and that's cool14:56
*** dasm is now known as dasm|afk15:02
inc0allright, out to home, laters15:03
*** inc0 has quit IRC15:04
openstackgerritMerged stackforge/kolla: Check compose cmd result  https://review.openstack.org/17712215:04
sdake_samyaple agree on the cumbersome part15:05
sdake_samyaple i'm thinking the proper word is "slow" :)15:05
sdake_it creates a new workflow for creating containers which i don't particuarly like15:06
sdake_inc0 sdake.io15:15
sdake_read it ;)15:15
sdake_new toy http://www.stevehuffphoto.com/line-magnetic-219ia-integrated-tube-amp-review-300b-and-845-tube-magic/ on its way to phoenix as we speak15:17
sdake_should have by friday ;)15:17
sdake_i a/b blind compared to a 45k tube amp, couldn't tell the difference15:20
sdake_with a 100k usd preamp+phono stage15:20
*** mattt has joined #kolla15:20
pdbI think the above sounds good on paper, until you tell an admin he has to wait > a few minutes just to turn on verbose in a service :/15:20
sdake_pdb you mean the data volumes for /etc?15:20
pdbyes15:21
sdake_that is the slow part i was referring to15:21
sdake_the cumbersome part -we can abstract that away15:21
pdbas in hide it?15:22
sdake_right15:23
sdake_but performance can't be abstracted15:23
sdake_performance is what it is ;)15:23
pdbjust what I was about to say15:23
sdake_actually that isn't technically accurate15:24
sdake_some of it could be abstractedby building a million data containers with all possible options :)15:24
sdake_but this comes with a new set of problems which are worse on balance imo :)15:25
sdake_where million is some large number not actually 1 milion :)15:25
pdb:P15:25
pdbI feel it comes back to trying to not let the user screw up their system15:26
pdbPersonally I think that's envitable, not sure I'd sacrifice usability just to slow down a cowboy admin ;)15:27
pdbisn't that why python dont have private variables?15:27
pdb*doesn't15:28
bmaceit can and will happen, no matter how much you try to protect them.  once you make something stupid proof, stupid just gets stronger :)15:28
*** absubram has joined #kolla15:28
bmacehaving stuff mounted from /etc in the sort of "regular" place does, on the upside, give admins a very similar feel to a traditional openstack deploy so if they have existing tools / procedures more of it seems likely to work for that scenario.15:31
bmacei appreciate the increased lines of code needed to handle both ways.  i think sam suggested double, but if that is from 10 lines to 20 lines that isn't too much of a price to pay imo.  1000 to 2000 would be a bit rougher, but i would be a little surprised if it was that much.15:32
SamYapleso a data container with the option of bindmounting is no good? I think that solves the issue nicely15:35
SamYaplei should clarify15:37
pdbSamYaple: no I think that might work15:37
sdake_exceptions are a way to the dark side15:37
SamYaplebindmounting for the datacontainer only15:37
sdake_convince me this isn't an exception15:37
SamYapleconvince me this is an exception?15:38
pdbsdake_: I could say the same about crudini15:38
pdb?15:38
sdake_you mean crudini and serialization is an exception15:38
sdake_let me address that point, I struggled with this as well15:38
sdake_it is in Kolla's best interest to have our containers used by as many downstreams as possible15:39
sdake_this model enables that behavior15:39
sdake_hence it facilitates our mission15:39
sdake_it makes us stronger15:39
sdake_https://www.google.com/search?es_sm=119&q=define+exception&oq=define+exception&gs_l=serp.3..35i39j0i67j0i20j0l7.3491.3661.0.4207.2.2.0.0.0.0.363.363.3-1.1.0....0...1c.1.64.serp..1.1.362.DQFc8WFGyVA15:40
sdake_it does not result in exclusion it resultts in inclusion15:40
bmacethe concern is that for some reason doing the bindmount of the config file is somehow less variable / non-immutable than passing in a bunch of env data that has the same effect on the process in the container?15:41
sdake_the problem bmace is the data can *change* on the host15:41
bmaceit is just two different ways to get exactly the same data into the container, and i think to your point, having the additional way is more inclusive.15:41
sdake_this is where Kevin Fox's idea comes in15:41
sdake_which I think I support and is not an exception :)15:42
*** sdake_ has quit IRC15:42
*** fabiand has joined #kolla15:43
pdbthe theme in openstack lately seems to be if there's enough people wiling to support something it's a viable option15:45
bmaceso most of your concern is, specifically, around config changes underneath a running container?  do we have some containers that actually read that stuff post execution?  i appreciate to some degree we are concerned about container lifecycle, but it is sort of an obnoxious limitation to say all config changes must be made through a kolla master, which then does all the env push stuff, etc.  it greatly limits the freedom / control of the15:45
bmaceostk admins.15:45
pdbthere seems to be quite a few who would like to see the bind mounts as an option15:45
*** sdake has joined #kolla15:47
*** sdake_ has joined #kolla15:48
sdake_bmace power dropped off15:49
sdake_needed to find a plug15:49
sdake_where I lost power : [08:41:47]  <sdake_>this is where Kevin Fox's idea comes in15:49
sdake_[08:42:04]  <sdake_>which I think I support and is not an exception :)15:49
sdake_samyaple seemed to indicate it was dancing on the razor blade of what is immutable15:49
openstackgerritMerged stackforge/kolla: Make kolla script bashate compliant  https://review.openstack.org/18643315:50
openstackgerritMerged stackforge/kolla: Make setup_docker.sh bashate compliant  https://review.openstack.org/18643415:50
openstackgerritMerged stackforge/kolla: Make update-build-links bashate compliant  https://review.openstack.org/18643515:50
bmacesure, one can change at start time, one can start at any time.. i'll toss you what little you missed in a PM since everyone else saw it already :)15:50
openstackgerritMerged stackforge/kolla: Make magnum demo start bashate compliant  https://review.openstack.org/18643715:50
pdbI was basically saying I think if enough people want it and are willing to support something, that thing can be made an option. difference between an option and exception...? um...15:50
openstackgerritMerged stackforge/kolla: Remove 1000 bashate failures by ignoring .git directory  https://review.openstack.org/18643815:50
sdake_would appreciate it so i don't have to hunt down the logs15:50
*** sdake has quit IRC15:52
sdake_[08:51:00]  <bmace>so most of your concern is, specifically, around config changes underneath a running container?  do we have some containers that actually read that stuff post execution?  i appreciate to some degree we are concerned about container lifecycle, but it is sort of an obnoxious limitation to say all config changes must be made through a kolla master, which then does all the env push stuff, etc.  it greatly limit15:53
sdake_yes containers read the data post execution15:53
sdake_on restart15:53
sdake_kevin fox's proposal is to fix that so that doesn't happen15:53
*** jasonsb has joined #kolla15:54
sdake_by copying a bindmounted /etc dir on start into the system, then removing the bindmount15:54
pdbI would read kevin's comment as wanting bind mounts :)15:54
SamYaplesdake_: you cant remove bind mounts like that15:54
sdake_its not the bindmount I have allergy to, its the changing of the container during runtime15:54
bmaceif someone has some setting they want to change on a compute node, and they know that there are no vms on it.. or even that it is safe to do it in the state that they are in at that point.. then i say they should be within their rights to change the config the way they always have, and restart the container.15:54
sdake_sorry removing the bindmounting file - let me clarify samyaple15:54
SamYapleand /win 115:55
sdake_then we end with two sources of truth bmace15:55
bmaceif it is only @restart that the changes are picked up though, it isn't at runtime.15:55
bmacewell, it isn't "during" runtime :)15:55
*** sdake_ is now known as sdake15:56
bmacesure.. the truth thing, yeah, we had gotten around that by markup in the config file, like a gui builder would do15:56
bmacewe had fields that were marked up so the user knew that it would be overwritten if pushed out by the management process15:56
*** daneyon has joined #kolla15:56
sdakemorning daneyon15:56
sdakehey i failed to register for cl in time15:56
sdakegot a bone for me to solve that problem15:57
sdakecharles forwarded me on to someone else who didn't respond :(15:57
sdakeone a container starts, it should enter a immutable state imo15:57
sdakeotherwise it looses its declartive property15:58
*** jtriley has quit IRC15:58
sdakeI am open to Kevin's idea if the wider kolla community accepts it15:59
daneyonmorning sdake15:59
daneyonnot sure what i can do to help. let me look into it and see if anything can be done.15:59
sdakethe idea being bindmount a file in /etc, cp it in the container, delete it on the host from the container16:00
*** jtriley has joined #kolla16:00
sdakesamyaple what are your thoughts on this approach?16:00
bmaceas long as it doesn't actually re-read that file after start time imo the behavior doesn't really seem much different than pushing in env vars.. if it picked up changes on the fly, not just container restart i could see that would be odd / dangerous.16:01
sdakeright when people talk about bindmounting etc, its the re-read that concerns me16:01
sdakei don't actually care much about the bindmount ;)16:01
*** jasonsb has quit IRC16:02
sdakei think from an implementation perspective it could be racy - too bad there isn't an atomic cp&rm :)16:03
sdakeI'm not clear if mv would work in a bindmount situation16:03
*** jasonsb has joined #kolla16:03
sdakedaneyon perhaps you can ping rohit and see what my options are?16:03
sdakecharles mentioned onsite registration16:03
sdakejust need to make sure i can get a explorer pass onsite16:04
sdakepretty sure patrick would freak at a 2k cl expense pass :)16:04
daneyoni think you will be fine with getting your pass onsite. We'll make it happen one way or another. Def no need to get a 2k pass16:05
sdakethanks that takes a bit of worry away :)16:05
sdakei arrive tuesday 9am leave wednesday 7pm16:05
daneyoni'm heading down this afternoon to avoid traffic between 4-716:06
daneyonI need to finalize creating a demo for the Netplugin project to help Vipin and team.16:07
*** jasonsb has quit IRC16:08
daneyonsdake, so does this online registration site not work for you: https://www.ciscolive2015.com/portal/startNewRegistration.do16:09
sdakeit says registration is closed16:09
daneyonsdake what if you call the support number on that site?16:09
SamYaplesdake: i dont like deleting the host. I dont like making the "re-read" a requirement for the project. I think the default should be immutable, but the option to be able to update the config and have the container pick up those changes should be up to the informed user16:09
sdakeoh snap16:09
sdakeit looks like its open again :)16:10
sdakelet me try again16:10
daneyonOK16:10
SamYaplehey daneyon16:10
daneyonhey SamYaple16:10
sdakesamyaple while I dislike taking choices away from the operator, I dislike two sources of truth more16:11
sdakelesser of two evils and all that ;)16:11
*** daneyon_ has joined #kolla16:12
SamYaplethe point pdb brought up is a very valid one. new users and admins want to be able to add an option to a config and test it out on one host16:12
SamYaplereguiring ansible for that is like using tnt to open a door16:12
*** daneyon has quit IRC16:14
sdakedaneyon do you know if there is a social event tuesday night?16:19
sdakedaneyon nevermind found it16:21
sdakei'll make my own social event :)16:21
sdakeagree with pdb's point, to that i'd recommend a different approach which is a production vs testing deployment16:26
SamYaplestill the same issue, testing or no16:27
sdakewould we want production deployments mucking with one node's config?16:27
SamYapleas an admin, thats sorta my job. i do it on a daily basis16:28
SamYapleso yes, thats the answer if it wasn't clear16:29
*** rhallisey|pto has quit IRC16:31
openstackgerritJeff Peeler proposed stackforge/kolla: Make config-glance.sh bashate compliant  https://review.openstack.org/18646316:34
sdakejpeeler did I miss one?16:34
jpeelernot sure, it said it needed rebasing so i just clicked the button16:34
sdakeoh a rebase16:34
sdakegot it16:34
sdakewonder how that happened16:35
sdakei rebased before i submitted sunday the patch stream with the comments fixed16:35
jpeeleronce that one goes, i think the rest will follow16:35
sdakesince its a rebase and already has two +2's, its good for a workflow +116:36
jpeeleryeah, about to do that now16:36
sdakesamyaple what if the deployment didn't need mucking with :)16:37
sdakeI don't want people to have to be experts on how kolla operates to do their jobs16:37
sdakeor containers in general16:37
sdakeor even openstack config options16:37
sdakeis the use case you want to test out a new feature on one node but not all nodes?16:38
SamYapleperfect worlds are perfect. but you are requiring people to be expects to change a config on one node for testing16:38
SamYapleive got to disagree on this one. my point in bringing it up was that I am not the only one who wants it and this discussion would be had again16:38
sdakei haven't drawn any lines in the sand :)16:39
sdakei am listening16:39
sdakewhat I want to understand is the use case16:39
sdakewhy would an admin want to change the config on one node and not all nodes16:39
sdakeyou do it on a daily basis, why?16:40
SamYapleuse case: I want to change a config option without having to rerun ansible for testing16:40
sdakeso you want to test one specific config option in a production cluster?16:41
SamYapledaily basis is anything from turning on debug+verbose for a single service, to adding in some wonky nova option to see if it fixes an issue16:41
SamYaplenormally i do this 1: test in lab (if possible) 2: test on one node in environment to verify it works 3: deploy env wide16:41
sdakecanary deployment16:41
SamYapleif there is a name for it?16:42
sdakecanary = automated16:42
sdakecanary deployment = make config changes in one place, if all is good, make in more places, if all is good, make in all the places16:42
SamYaplehow does it relate16:42
SamYapleok16:43
*** rhallisey|pto has joined #kolla16:43
sdakei absolutely want to get there16:43
SamYaplefor the record, these is no process that will work _better_ and be more accepted than tweaking a config on the host and running `docker restart <container>`16:43
sdakei think it should be automated - sounds like you think it should be manual16:43
SamYapleyou method exludes mine, mine works in conjunction with yours16:44
sdakethe canary method doesn't rule out manual config changes16:44
sdakeif b works in conjunction with a, a works in conjunction with b :)16:45
SamYaplealright. more succinctly. You say you don't want to use a method so it should be excluded. I say it should be included if it doesn't break your method/add overhead16:46
SamYaplewhich it doesnt16:46
sdakei just want one source of truth16:47
*** jtriley has quit IRC16:47
sdakeif the admin changes something there are two sources of truth16:47
sdake(rather changes something in two places)16:47
SamYaplethe one source of truth is the config file.16:47
sdakeadmin A comes along and changes X, admin b coms along and rechanges X a different way in a different place16:47
sdakei thought the one source of truth is the ansible augmentation and overrides file :)16:48
SamYaplei get it. i disagree. but i get it16:48
SamYaplei think bindmounting and changing the config from the host should be an option, not the default, but an option16:48
SamYapleI can almost gauruntee it would be used by everyone too16:48
SamYapleat least any admin that gets a hold of it16:49
sdakeok i am good with options16:49
pdb+1 on options16:49
sdakeas long as it isn't the default and there is a big ass warning say "please don't do this, we warned you so" :)16:49
pdbcall it "test mode" or something16:49
SamYaplethat I am ok with (and suggested earlier)16:49
sdakesamyaple apologies i lost poer so lost some of the context of the convo16:50
SamYaplei would prefer a documentation section on the section dedicated to the pitfalls of the bindmount method16:50
pdbthere's still a conversation to be had around the config merging mechanism. which im not convinced is necessary16:51
pdbbut might have to leave that for tomorrow16:51
sdakepdb not necessary for which reason16:51
SamYaplepdb: that i need ot hear about though16:51
sdakepdb ok enjoy your rest itme :)16:51
sdakepdb please put comments in the review16:51
sdakeso poeple don't have to track down our arguments 6 mo from now on the irc logs :)16:51
pdbmakes sense16:51
SamYaplebindmounting would require a config container though16:52
sdakethe config merging is to support one of our core use cases though, which is custom configuration16:52
sdakeand that is necessary ;)16:52
sdakethat is one thing that makes kolla significantly different from every other deployment tool in existence16:52
harmwhi guys16:53
sdakehey harmw welcome to the party!16:53
SamYaplefor the record sdake, when done correctly the bindmounting or "test mode" can be turned on and off through ansible at the cost of restarting all containers16:54
*** vinkman has quit IRC16:54
sdakecool like i said, i'm good with options16:54
SamYaple*recreating all the containers16:54
sdakeas long as the options that lead to disaster for a deployment are well documented :)16:54
*** jtriley has joined #kolla16:55
*** vinkman has joined #kolla16:55
*** dwalsh has quit IRC16:56
harmwjpeeler: missing openssl... bummer16:58
sdakesamyaple the tesst mode bind mounting on off, ike that idea, that way operators can force their admins into their policies based upon our fantastic documentaed warnings :)17:00
SamYaplei only call it test mode for your sake. i think its a valid run mode, when aware of the risks17:01
sdakeok well we should rename it something we can all agree on17:02
sdakerather then test mode - i dont like that name either17:02
SamYaple"hard mode"17:02
SamYaple"old school mode"17:02
SamYaple"cowboy mode"17:02
sdakebrings back old memories of world of warcraft ;)17:02
SamYapleprobably "legacy mode"17:02
sdakei like cowboy mode ;)17:02
sdakeshows we have a sense of humor ;)17:03
SamYaplei like legacy. cowboy still implies unsuitable for production17:03
bmaceunprotected? :)17:04
SamYaplelegacy says it is an older, less recommended way of doing things.17:04
vinkmanSo, exposing my total inexperience with Ansible, can Ansible not target a specific server for config changes, is it an all or nothing?17:05
SamYaplethats better than "advanced" or similiar which implies more skill/knowledge is needed17:05
*** daneyon has joined #kolla17:05
SamYaplevinkman: yes and no. it _can_ but if it is a single node it cant use info from other nodes17:05
SamYapleso in the case of haproxy you cant make a change to one haproxy node without touching ALL the nodes17:06
SamYaplebecause haproxy uses info about each node (ip addresses and what not)17:06
SamYapleI would strongly not recommend it for single changes to configs17:06
*** daneyon_ has quit IRC17:07
vinkmanBut it is possible to do that for the testing case?17:07
sdakesamyaple that is a real bummer I had hoped that would be possible to implement canary deployment17:07
*** dims has joined #kolla17:07
SamYapleit can probably be worked in, but in a very non-ansible type of way and 1bajjillion skpped messages17:08
vinkmanI think in general config changes that are node specific would be compute only not really on the controller side of the house...17:08
SamYaplevinkman: think admin use. testing change to an api server that is out of the LB pool17:09
SamYaplepermenant changes would be applied through ansible anyway17:09
*** dims_ has quit IRC17:10
*** bradjones has quit IRC17:10
*** bradjones has joined #kolla17:11
*** bradjones has joined #kolla17:11
sdakevinkman if someone wants changes on compute they will want em on controller in the same wy they are accustomed to17:12
SamYaplesdake: i never put [libvirt] changes on the controller or [database] on the compute. what do you mean?17:13
*** jmccarthy has joined #kolla17:13
sdakewhile we have an active chat going, can folks have a vote on our manifesto - line 517:17
sdakehttps://etherpad.openstack.org/p/kolla-manifesto17:17
*** rhallisey|pto has quit IRC17:19
openstackgerritMerged stackforge/kolla: Make config-glance.sh bashate compliant  https://review.openstack.org/18646317:22
harmwsdake: done17:25
harmwonly L5? or rest as well?17:25
harmw(though I'm seeing quite a lot +1 on the other topics already)17:26
sdakeharmw which other topics17:26
sdakeyou mean down the screen17:26
harmwyup17:26
sdakethat was our collaborative "make it yourself" model17:26
harmwhehe17:26
sdakethose were merged into one17:26
*** pradk has joined #kolla17:30
*** rhallisey|pto has joined #kolla17:30
sdakerhallisey|pto if your not really on pto plz vote on https://etherpad.openstack.org/p/kolla-manifesto17:32
*** jasonsb has joined #kolla17:33
*** daneyon_ has joined #kolla17:45
*** daneyon has quit IRC17:48
*** mstachow has joined #kolla17:50
mstachowhi :)17:50
*** weiyu-001 has quit IRC17:52
*** weiyu-001 has joined #kolla17:52
sdakedaneyon can you send me your cisco lie kolla slides - curious to see what you came up with17:54
sdakemstachow morning fine sir check out https://etherpad.openstack.org/p/kolla-manifesto and vote plz - leae comments if you like - line #517:55
mstachowhuh no problem :)17:56
sdakedont edit statement directly plz17:57
sdakeanyone else that plans to contribute to kolla over the liberty cycle is welcome to vote as well - enjoy ;)17:58
sdakediga u awake at this hour17:59
sdakeprad your welcome to vote as well ;)17:59
openstackgerritSteven Dake proposed stackforge/kolla: Make config-neutron.sh bashate compliant  https://review.openstack.org/18649018:01
openstackgerritSteven Dake proposed stackforge/kolla: Make get-image.sh bashate compliant  https://review.openstack.org/18915618:01
openstackgerritSteven Dake proposed stackforge/kolla: Make config-l3-agent.sh bashate compliant  https://review.openstack.org/18649118:01
openstackgerritSteven Dake proposed stackforge/kolla: Make config-linuxbridge-agent.sh bashate compliant  https://review.openstack.org/18649218:02
openstackgerritSteven Dake proposed stackforge/kolla: Make config-sudoers.sh bashate compliant  https://review.openstack.org/18649318:02
openstackgerritSteven Dake proposed stackforge/kolla: Make config-dhcp-agent.sh bashate compliant  https://review.openstack.org/18649418:02
openstackgerritSteven Dake proposed stackforge/kolla: Make neutron-server start.sh bashate compliant  https://review.openstack.org/18649518:02
openstackgerritSteven Dake proposed stackforge/kolla: Make swift bashate compliant  https://review.openstack.org/18916618:02
openstackgerritSteven Dake proposed stackforge/kolla: Make config-nova-compute bashate compliant  https://review.openstack.org/18648218:02
openstackgerritSteven Dake proposed stackforge/kolla: Make mariadb config-mysql.sh bashate compliant  https://review.openstack.org/18647518:02
openstackgerritSteven Dake proposed stackforge/kolla: Make mysql-entrypoint.sh bashate compliant  https://review.openstack.org/18647718:02
openstackgerritSteven Dake proposed stackforge/kolla: Make nova-controller start.sh bashate compliant  https://review.openstack.org/18649918:02
openstackgerritSteven Dake proposed stackforge/kolla: Make keystone start.sh bashate compliant  https://review.openstack.org/18647618:02
openstackgerritSteven Dake proposed stackforge/kolla: Make config-nova.sh pass bashate gate  https://review.openstack.org/18646518:02
openstackgerritSteven Dake proposed stackforge/kolla: Ignore .tox directory to remove some bashate failures  https://review.openstack.org/18646418:02
openstackgerritSteven Dake proposed stackforge/kolla: Make rabbitmq start.sh bashate compliant  https://review.openstack.org/18646718:02
openstackgerritSteven Dake proposed stackforge/kolla: Make nova-libvirt's start.sh bashate compliant  https://review.openstack.org/18646618:02
*** sdake has quit IRC18:02
*** sdake has joined #kolla18:02
sdakegroan18:03
sdakejenkins got busted18:03
pradksdake, cool, will take a look and vote.. thx for the heads up18:04
jpeelerharmw: yeah, silly problem, but worth fixing18:07
harmwtotally :)18:07
harmwjust didn't expect openssl would be unavailable on some systems :)18:08
jpeeleri'm using the fedora cloud image18:08
harmwhmk, oh well18:08
*** openstackgerrit has quit IRC18:09
harmwIll work out a little patch later tonight :)18:09
*** openstackgerrit has joined #kolla18:09
harmw*hopefully18:09
jpeelerno rush18:09
*** dwalsh has joined #kolla18:12
*** rhallisey|pto has quit IRC18:15
*** jruano has joined #kolla18:15
*** rhallisey has joined #kolla18:16
*** shardy_ has joined #kolla18:21
*** shardy has quit IRC18:22
sdakegot job spam for https://www.linkedin.com/jobs2/view/54419099?trk=eml-jymbii-organic-job-title&refId=df805575-8209-40c9-b947-0b508136b74b&midToken=AQEm0QXe70O0UA18:23
sdakeI must be radioactive ;)18:23
openstackgerritMichal Rostecki proposed stackforge/kolla: [WIP] Add designate-sink service  https://review.openstack.org/18939318:24
*** shardy_ has quit IRC18:26
*** daneyon_ has quit IRC18:26
*** shardy has joined #kolla18:27
sdakehey shardy18:27
harmwoh my, someone trying to add Sink :)18:28
harmwnice18:28
*** mstachow has quit IRC18:28
*** mstachow has joined #kolla18:29
*** rhallisey is now known as rhallisey|pto18:32
nihiliferyes, I really need containerized sink :)18:33
harmwI'm already reviewing :)18:35
*** jtriley has quit IRC18:45
sdakebest part of empire strikes back18:45
sdake"Try?  Try Not.  Do or do not.  There is no try."18:45
harmwhehehe18:46
*** jtriley has joined #kolla18:49
*** loth has joined #kolla18:54
openstackgerritMerged stackforge/kolla: Ignore .tox directory to remove some bashate failures  https://review.openstack.org/18646419:08
*** prad__ has joined #kolla19:14
*** fabiand has quit IRC19:18
openstackgerritMerged stackforge/kolla: Make config-nova.sh pass bashate gate  https://review.openstack.org/18646519:18
*** prad__ has quit IRC19:23
*** pradk has quit IRC19:28
harmwnihilifer: you got Sink working though?19:31
openstackgerritMerged stackforge/kolla: Make nova-libvirt's start.sh bashate compliant  https://review.openstack.org/18646619:42
openstackgerritMerged stackforge/kolla: Make rabbitmq start.sh bashate compliant  https://review.openstack.org/18646719:42
openstackgerritMerged stackforge/kolla: Make mariadb config-mysql.sh bashate compliant  https://review.openstack.org/18647519:43
openstackgerritMerged stackforge/kolla: Make keystone start.sh bashate compliant  https://review.openstack.org/18647619:43
openstackgerritMerged stackforge/kolla: Make mysql-entrypoint.sh bashate compliant  https://review.openstack.org/18647719:43
openstackgerritMerged stackforge/kolla: Make config-nova-compute bashate compliant  https://review.openstack.org/18648219:43
openstackgerritMerged stackforge/kolla: Make config-neutron.sh bashate compliant  https://review.openstack.org/18649019:45
openstackgerritMerged stackforge/kolla: Make config-l3-agent.sh bashate compliant  https://review.openstack.org/18649119:45
sdakebatehate coming to a conclusion yay :)19:45
openstackgerritMerged stackforge/kolla: Make config-linuxbridge-agent.sh bashate compliant  https://review.openstack.org/18649219:46
openstackgerritMerged stackforge/kolla: Make config-sudoers.sh bashate compliant  https://review.openstack.org/18649319:46
openstackgerritMerged stackforge/kolla: Make config-dhcp-agent.sh bashate compliant  https://review.openstack.org/18649419:46
openstackgerritMerged stackforge/kolla: Make neutron-server start.sh bashate compliant  https://review.openstack.org/18649519:46
openstackgerritMerged stackforge/kolla: Make nova-controller start.sh bashate compliant  https://review.openstack.org/18649919:47
jpeelerlooks like cinder needs fixing19:48
sdakewaiting on the cinder merge19:49
sdakeand then I think its job done :)19:49
jpeeleronce we get everything in, need to make it voting19:50
sdakeagree19:50
sdakeprobably about time to make the functional gate voting as well19:51
sdakeseems to work well enough19:51
jpeelerwonder if it's a no no to enable both at the same time19:52
*** bmace has quit IRC19:56
harmwsdake: https://review.openstack.org/#/c/187981/3 your thoughts please20:02
harmwI don't want this to end up in some childish (no offence to anyone here) yes vs no btween just 2 persons :)20:03
*** bmace has joined #kolla20:07
openstackgerritMichal Stachowski proposed stackforge/kolla: WIP: Galera container  https://review.openstack.org/18722520:08
sdakeharmw groan reviewer fatigue20:08
sdakegive me 15 minutes to get the monster into my bloodstream20:08
harmw:P20:08
mstachowgood night :)!20:14
harmwmstachow: thank for throwing in new stuff to review :)20:16
*** mstachow has quit IRC20:19
sdakeharms lgtm for the most part20:26
harmwok, specifically the single node stuff about hostnames and static id's20:27
harmwam I simply wrong on those?20:27
sdakejust difference of opinion20:28
sdakeother reviewers may have other opinions20:28
harmwyup20:28
sdakemy general rule of thumb is if it solves our problem i'm good with it20:28
sdakewhen it doesn't we can rework20:29
harmwtrue, while I'd like to get most stuff worked out up front20:29
harmwwhenever possible that is20:29
sdakei'm more focused on output then perfection :)20:30
harmw:P20:30
sdakethe hostname static id thing20:30
sdakenot sure on that20:30
sdakenot a keepalived expert20:30
sdakethat is why we require two core reviewers to review a patch20:31
sdakei don't even know what that stuff means ;)20:31
sdakebut you asked me to review so I did to the best of my ability20:31
sdake140 pep errors left to fix ;)20:31
sdakerather bashate errors20:31
sdakeyay for progress20:31
harmwmy point (in general here) is he's building this to be run on specifically a single dockerhost, wich I'm opposed to since I'd rather have the ability to run a 2nd keepalived to do $whatever (even if that might not even happen anytime soon)20:32
harmwbut as I said, I could be striving for the wrong thing here20:32
harmwjust want to hear that from someone  :)20:33
*** shardy_ has joined #kolla20:34
*** shardy has quit IRC20:36
sdakewell its clear we want keepalive20:37
sdakethe question is do we want two keepalive on same node or not20:37
harmwofcourse20:37
sdakei dont see a use case for that20:37
sdakebut maybe a different core reviewer does20:37
sdakei've seen stuff hit our repo with four -1 votes before ;)20:37
harmwdepends, do we want 1 keepalived per dockerhost to handle everything, or do we wish for multiple? What about using dockerhost-1 for both a production and a testsetup? that would make it have 2 keepalived containers, using (though perhaps not sharing) atleast the same hostname...20:38
harmwbut yes, I'm intrested to see what others think :)20:39
sdakejust tyring to keep it simple i think mixing dev and prod is a bad idea :)20:39
harmwofcourse it is :p20:39
sdakeand using bad use cases to justify a rationale is not ideal :)20:40
*** shardy_ has quit IRC20:40
harmwbut should things break if $operators decides to go that route?20:40
sdakehopefully operators wont have to do that - itwill be automated20:40
*** shardy has joined #kolla20:40
sdakewe know docker-proxy is bad for performance wich means only 1 of our containers can run per node in general20:40
sdakei dont need how you would get two nova-apis on the same node for example20:41
harmwofcourse20:41
harmwbut do we want 1 keepalived for everything, or perhaps a 2nd keepalived for just service X?20:42
harmw(1 as in 1 per dockerhost)20:42
*** dwalsh has quit IRC20:42
sdakei think that decision is best left to daneyon20:42
sdakehe is the individual leading our ha spec effort20:42
sdakedid you see his ha spec?20:43
harmwI'd rather have it solid per design, rather then because we choose to configure it in a specific way20:43
sdakethis is a question to ask the experts in that area20:43
harmwgood one, I read it a while ago20:43
harmwperhaps I should revisit20:43
sdakehttps://review.openstack.org/#/c/181983/20:43
harmwand I should commnt on some multihost spec as wel20:44
sdakeplease add your comments there re keepalive on multihost20:44
sdake(in the ha spec)20:44
harmwyep, will do20:44
sdakeI assume michal is working based upon that draft spec20:44
sdakeharmw you do good reviews - not sure how you pick out all these details ;)20:46
sdakere galera implementation20:46
harmwdetails?20:47
sdakecheck_vars for example20:47
sdakemy brain implodes every time i do a new container review20:47
sdakedesignate took a double monster to get through ;)20:47
harmwhehe20:47
harmwthough I hate this last review, since I'm reading stuff now that I should (and could) have adressed in my first review :p20:48
sdakethat is reviewer fatigue20:49
sdakethat happens on bit commits20:49
sdakeunfortuantely oru containers are big20:49
harmwcinder was big20:49
harmwgalera is relatively easy20:49
harmwanyway, bedtime :p20:50
sdake   loth are you about20:56
*** pradk has joined #kolla20:57
lothWhats up20:58
sdakeloth could you vote on line #5 of https://etherpad.openstack.org/p/kolla-manifesto21:02
sdakesemi-final wording of our manifesto21:02
lothOk21:04
sdakethanks loth21:07
lothnp21:08
lothworking on a baremetal ansible deployment to get familar with everything21:09
*** sdake_ has joined #kolla21:19
*** vinkman has quit IRC21:21
*** sdake has quit IRC21:23
*** vinkman has joined #kolla21:24
*** vinkman1 has joined #kolla21:28
*** vinkman has quit IRC21:28
*** jtriley has quit IRC21:29
*** jruano has quit IRC21:38
jpeelerquick sanity check: if i do "docker images" and then do docker-compose for a specific service, the created column should match, right21:52
jpeeleri had to specify the image sha in the compose file. i must be doing something wrong... end of day now though22:02
*** pradk has quit IRC22:08
sdake_use build --release jpeeler22:14
sdake_--release wlil tag it with latest22:15
sdake_vs its sha22:15
*** absubram has quit IRC22:38
*** dwalsh has joined #kolla22:59
*** bmace_ has joined #kolla23:01
*** bmace has quit IRC23:05
*** daneyon has joined #kolla23:10
*** daneyon_ has joined #kolla23:11
*** daneyon has quit IRC23:15
*** dwalsh has quit IRC23:24
*** bmace_ is now known as bmace23:28

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!