15:59:59 #startmeeting Fuel 16:00:00 Meeting started Thu Sep 4 15:59:59 2014 UTC and is due to finish in 60 minutes. The chair is vkozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:01 hi 16:00:02 hello 16:00:03 The meeting name has been set to 'fuel' 16:00:05 hi 16:00:09 #chair vkozhukalov 16:00:10 Current chairs: vkozhukalov 16:00:12 hi 16:00:13 seems like many people here, good 16:00:21 yes 16:00:25 agenda is huge, pls be prepared 16:00:51 #topic Patching of OpenStack: current status, remaining issues. 16:01:02 me is talking 16:01:09 hi all 16:01:13 agenda as usual is here https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda 16:01:16 hello all 16:01:17 vkozhukalov: I think you can also copy the guy in brackets, who will be providing ingo 16:01:33 mihgen: ok 16:01:45 hi 16:01:52 aglarendil: your turn broadcast 16:01:54 hi all 16:02:12 we managed to write a hook that applies before patching starts 16:02:32 it removes all openstack packages and allows puppet to redeploy all the openstack code 16:03:05 also, we need to apply some patches to orchestrator and nailgun part to run patching one node by one node 16:03:17 and allow user to restart patching, not only to rollback 16:03:27 so, currently we are facing some issues with rollback 16:03:49 1) some of swift manifests need to be backported to old 5.0 to allow rollback 16:04:12 2) I am seeing some issues with pacemaker issues right now, so we need to add some commands to pre patching hook 16:04:20 my estimate that it will take one more day 16:05:11 what about missing version.yaml and lack of info in api/version? was that about patching? 16:05:38 I guess, no 16:05:52 aglarendil: ok, thanx, sounds well 16:05:58 moving on 16:06:09 #topic Juno testing update (mattymo) 16:06:23 thanks vkozhukalov 16:06:33 I've been working with Timur Nurlygayanov, Denis Egorenko, and Ivan Berezovskiy on Juno readiness. The first goal, which was accomplished last week, was to get Fuel Master fully functional under Juno packages. 16:06:41 The three main blockers were hardcoded python version dependencies for nailgun and ostf, keystoneclient missing dependencies, and saharaclient dependencies. I've worked around these for now in https://review.openstack.org/#/c/117801/ until they are all fixed. 16:06:52 Fuel Master deploys fine. OpenStack deploys and starts all services fine. There are still some dependency issues preventing horizon from working, but we can provision instances now. 16:06:58 Lots of patches and bugs are open that are waiting to be merged after freeze here: https://etherpad.openstack.org/p/fuel-6.0-update The majority of changes are small Puppet config changes and rebasing or new python dependencies. 16:07:37 mattymo: wow that sounds like very good progress 16:07:43 Also, all bugs related to this effort are tagged with "juno". 16:07:44 can you provide OS / modes you tested? 16:07:53 CentOS simple and HA. Ubuntu simple. 16:08:03 what about ceph, and other ? 16:08:05 neutron hasn't been tested yet. Most effort was on ceilometer and horizon today. 16:08:17 Still haven't tested sahara or heat, as far as I understand. 16:08:20 oh ok, neutron was always pain 16:08:29 murano deploys heatr 16:08:39 so if you tested murano then you covered heat 16:08:49 thanks mattymo, very good progress 16:08:55 moving on 16:08:58 I don't think we did murano yet either. I'm more or less putting out fires as they go along. 16:09:04 vkozhukalov, go ahead 16:09:11 #topic Ensure that Fuel can reliably provision at least 100 nodes - status? (sambork) (holser) 16:09:18 mattymo: thanx 16:09:33 we started working with Lukasz Oles on bp for this task. In middle time we modified the load test writen by Alexander , we added profiler as a middleweare to collect stats and identify more bootlenecks, also we added new tests. We still need to contact mos openstack team to talk about this task. 16:09:51 sambork: can you provide bp for this? 16:10:04 we should target it to 6.0 sooner 16:10:24 tomorrow we will send bp to review 16:10:54 sambork: very good that you collaborated with akislitsky_ , hope we will be able to use Fuel CI for controlling further patches to nailgun 16:10:55 sambork: you mean spec? right? 16:10:56 I guess we’ll have more bottlenacks as operators, as our networking won’t be able to handle all virtual instances 16:11:01 any work done on astute side? 16:11:16 holser: it's more question to openstack itself, not fuel 16:11:22 should be targeted by mos-openstack team 16:11:33 our issue is provisioning of 100 nodes 16:11:53 PXE will die under if done simultaneously, astute should address sequences 16:12:01 vkozhukalov: yes i thought about spec 16:12:03 ok I think it should be discussed in spec 16:12:11 I’d say if we have separate network role, so we can use separate hardware for networking and that should help 16:12:26 let's move on, thanks for info, sambork - let's get bp tomorrow in specs 16:12:34 mihgen: it is not a problem as we can easily to substitute pxe with ipxe 16:12:36 regarding provisioning 100 nodes, hardware for fuel master node can be a big issue (especially disk IO), I'll add comments to BP 16:13:04 excellent, sambork - don't forget to send email to openstack-dev then once you put first version to review 16:13:13 #topic Extend authx Fuel feature to make sure we can upgrade 5.1 -> 5.1.1(sambork) 16:13:15 ok, np 16:14:03 Lukasz(salmon) is working on bp https://review.openstack.org/#/c/118284/1 so please comment and discuss it 16:14:14 is it about authx? 16:14:22 salmon around? 16:14:25 yep. good 16:14:37 no, he is away 16:14:39 evgeniyl_ is mandatory to review that 16:14:46 mihgen: got it 16:14:53 thx let's move on 16:15:08 #topic HTTPS for Fuel (prmtl) 16:15:22 Also I have a question, what do we do with not finished stage III from https://review.openstack.org/#/c/96429/11/specs/5.1/access-control-master-node.rst and not started stage IV . Should we focus also on this task? Because probably as a Polish team we will have enough resources 16:15:22 idea is to make all APIs to work under HTTPS, draft of the blueprint is here: https://etherpad.openstack.org/p/nailgun-ssl-endpoints 16:15:55 implementation has 3 stages: enable ssl, secure agent (need more research), user upload own certificate 16:15:56 it’s highly related to SSL for OpenStack endpoints since it should use the same certificate and mechanism for generating one 16:16:12 sambork: let's get this question in openstack-dev 16:16:33 mihgen: ok 16:16:41 prmtl: don't we want to combine it with SSL feature for OPenStack REST API endpoints ? 16:16:49 we can keep it separate too 16:17:09 but the part which is common should be in one blueprint /spec, like upload new cert into Fuel UI 16:17:22 SSL endpoints are terminated on controller, this is aobut our internal stuff 16:17:23 prmtl: is there a bp for it? 16:17:44 tzn: I understand it's about Fuel UI, correct? 16:17:50 afaik part about uploading own cert has been removed from the ssl feature for os api endpoint 16:17:56 UI, and possibly APIs 16:18:18 yes, that bp assumes self signed CA 16:18:25 s/CA/sert/ 16:18:31 prmtl: then let's get this removed part to new bp 16:18:52 mihgen: ok 16:18:53 we might want to discuss it over email too 16:19:04 let's not hesitate to spam ML ) 16:19:10 will do 16:19:17 prmtl: thanx 16:19:19 thanks folks, SSL is needed really 16:19:26 moving 16:19:36 #topic AuthX missing features 16:19:45 looks like duplicate topic 16:19:48 yes 16:19:50 was not this discussed 16:19:57 #topic API versioning status (nmarkov) 16:20:11 we had a couple of meetings on this 16:20:24 so, I’m currently working on a spec in fuel-specs 16:20:33 meow-nofer: excellent. link 16:20:34 ? 16:20:49 mihgen: not uploaded yet, planning for tomorrow 16:20:59 meow-nofer__: ok.. let's do this sooner 16:21:03 is it a blocker for 6.0? 16:21:13 most probably, yes 16:21:24 we need it at least in some ugly way 16:21:30 but if it's not fully ready, we could survive by workarounds, right? 16:21:38 like we did in 5.1 already? 16:21:47 mihgen: yep, I’ll try to do my best 16:22:05 sooner we implement it the better, the less to work around 16:22:11 let's be all prepared to review 16:22:20 need to continue with discussion of some related subjects 16:22:20 meow-nofer__ thanx 16:22:32 fuel-dev thread seems to be almost dead 16:22:35 #topic Granular deployment status (dshulyak) 16:22:41 first of all, there is spec, and i would appreciate reviews 16:22:41 https://review.openstack.org/#/c/113491/ 16:22:44 meow-nofer__: ping ppl there then 16:22:51 2. we have almost all code for iteration 1, i think warpc__ will finish tommorow 16:22:51 necessery astute refactoring, and it will enable end-to-end testing 16:22:55 3. tasklib already rewritten in python https://review.openstack.org/#/c/118311/, waiting for reviewers 16:23:00 4. there was some problems with dependencies for tasklib gem, but all 16:23:00 necessary stuff for python impl already in mirror, i will prepare small spec tommorow 16:23:19 dshulyak: great 16:23:27 dshulyak: looks like good progress for implementation, not just specing 16:23:32 i hope we will have working deployment by the end of next week 16:23:45 though bad that it goes too much forward from code ) 16:23:46 we need to review asap then 16:23:59 so there will be less need to rewrite some parts ) 16:24:10 dshulyak: excellent, thanks 16:24:21 does anybody want to review this spec& 16:24:23 ? 16:24:44 vkozhukalov: all nailgun/astute core should 16:24:58 actually it is related to library too 16:25:08 #topic Anonymous collection stats status (teran) 16:25:16 Currently we’re starting designing stats feature. 16:25:16 We’re already discussed some details and ready to start writing specs since next week. 16:25:16 Blueprint: https://blueprints.launchpad.net/fuel/+spec/send-anon-usage 16:26:17 teran: great 16:26:30 teran: good. I heard this gonna be led by akislitsky_ ? 16:26:30 any questions here? 16:26:44 mihgen, yep 16:26:47 mihgen: yep, he's a feature lead 16:26:55 so, moving on 16:27:05 ok thanks folks. Design has to be done by next Th 16:27:07 for 6.0 16:27:23 we need to get it running in 6.0, might be in some pre-production, but already working mode 16:27:31 #topic Image-based provisioning status (vkozhukalov) 16:27:53 this week I implemented bootloader installing stuff in fuel agent 16:27:58 it is under review 16:28:21 #link https://review.openstack.org/#/c/114245/ 16:28:27 vkozhukalov: important question :) When do we expect to enable all this? Can it become experimental in 6.0 ? 16:28:39 agordeev is now working on ceph stuff and functional tests 16:28:49 ceph?? 16:28:50 yes, it can 16:29:03 disk partitioning for ceph? 16:29:09 or what do you mean by ceph here? 16:29:47 yes, we need to add some logic which is still not implemented 16:29:55 vkozhukalov: ok, very good, would be great to see it in experimental soon in 6.0. 16:30:06 yes disk partitioning for ceph 16:30:20 actually some logic about ceph-journals 16:30:24 I would like to see pmanager die 16:30:34 we have tricky stuff in pmanager 16:30:42 i would too 16:30:43 xarses: +1 :) 16:30:48 moving on 16:31:04 #topic SSL support for OpenStack endpoints status(tuvenen) 16:31:10 First here is the link to the spec 16:31:10 #link https://review.openstack.org/#/c/102273/ 16:31:18 Status about the spec: The review is in good progress. Serveral people already 16:31:18 reviewed it. Basically I think that we are almost agree. 16:31:26 To give a quick reminder. We propose to configure the API endpoints on public 16:31:26 network with SSL. This will be done through HAProxy. That means that the 16:31:26 communication between the client and the HAProxy will use SSL. The decryption 16:31:26 will be done by the HAProxy and it will also pass the request to the 16:31:26 corresponding service. It works for Horizon as well. 16:31:37 As said we will use a self-signed certificate generated by astute and distributed on 16:31:38 all nodes that are running HAProxy. 16:31:45 There are still two points that are not clear in the spec: 16:31:51 First we suppose that HAProxy will always be deployed. So we added a link to 16:31:52 the single-controller-ha blueprint. We are not 100% sure if it will be in 16:31:52 Fuel 6.1. 16:32:03 And the other point is that currently we are only dealing with the self-signed 16:32:03 certificate. I think that the Rest API to manage the download of certificate 16:32:03 will be in another BP. It can be discussed directly in the spec review. 16:32:10 xarses: will we finally do it? I mean only HA? 16:32:25 tuvenen: yes, I think so 16:32:37 ok cool 16:32:43 About the code: 16:32:49 There are puppets manifests to sync+adapt the upstream manifest of haproxy and 16:32:49 to manage the configuration of HAProxy. Reviews are in progress and tests are 16:32:49 needed. It is not well tested because it requires to use HAProxy >= 1.5 and 16:32:49 it is not available for Fuel right now (the work is in progress). Fuel needs 16:32:49 special patches. 16:32:50 > can be another bp 16:32:51 +1 16:32:53 tuvenen: thanx, great status report 16:33:09 will read later today 16:33:09 and finally, The part about the management of the self-signed certificate is not started yet. 16:33:23 vkozhukalov, ok ) 16:33:35 moving on 16:33:38 tuvenen: fuel needs only one special patch 16:33:45 and it's very cleanly applicable 16:33:48 tuvenen, mihgen I can help with haproxy 1.5 16:33:56 tuvenen: thanks looks like we are in a very good shape with spec, though reviews are still needed 16:34:08 #topic multiple L2 networks (rmoe) 16:34:12 haproxy should be under msemenov's 16:34:14 angdraug, yes I think its almost ready 16:34:18 Currently all unit tests pass. BVT is still failing but the issues don't look related to my changes (CentOS fails with Paramiko issues and Ubuntu fails with swift-related errors). 16:34:28 xarses helped me set up a test environment with libvirt that is working now. That testing exposed a few small issues that I fixed yesterday. If the BVT tests will pass then I see no reason this code can't be merged now while I continue to test in our multiple network environemnt. 16:34:30 yes, Alex Mogylchenko is working on it now for 6.0 16:34:35 I almost had a successful deployment yesterday but I had some configuration issues in libvirt. I hope to have a successful end-to-end deployment today. 16:34:38 wait ubuntu should not fail 16:34:43 we should have green BVTs 16:34:46 aglarendil: are not we? 16:35:15 I was only able to run 2 yesterday (one was my fault and one was with some swift error) 16:35:25 the spec should also be updated, there's a -1 from dpyzhov related to upgrade impact 16:35:28 rmoe: excellent, do you get enough reviewers? 16:35:35 akasatkin: did you review that? 16:35:42 right now only akasatkin: has reviewd the changes 16:35:50 rmoe: and obviously this code is to drop once we open master 16:36:03 we need more pythonish force here 16:36:08 should be ready to go by then 16:36:10 yes, I have questions but I wait for review status of request 16:36:10 folks let's help with that 16:36:24 #link https://review.openstack.org/#/c/99179/ 16:36:30 There are three outstanding items right now: 16:36:35 1. Adding commands to fuel-cli to create node groups and assign nodes to them (this is in-progress right now) 16:36:40 2. Restrict the UI to only show networks from the default group. Right now the network settings screen will show ALL networks. 16:36:41 rmoe: there is a large docs piece I believe 16:36:47 3. Automatically manage DHCP ranges for additional fuelweb_admin networks. dnsmasq supports conf.d style configs so we can automatically write out the correct settings for each network and remove the files when networks are deleted. Users can also do this manually 16:36:57 yes, there will definitely need to be some doucmentation about this :) 16:37:18 what size of the env should be to start thinking about this feature? 16:37:20 50 nodes? 16:37:29 2 racks 16:37:31 could be even less than that 16:37:35 yeah 2 racks 16:37:54 ok. understood. but we can still survive with 100 nodes without this feature, correct? 16:38:04 but it's not gonna be scalable? 16:38:11 depends on the network hardware but yes it should still work 16:38:11 mihgen: yes, but barely 16:38:21 ok thx 16:38:45 #topic building deb packages when building iso  (brain461) 16:38:50 hello 16:38:52 Building of fuel deb packages during iso - code is ready, I'm testing it on my local box. Will ask our guys for review as soon as custom iso passes bvt. 16:39:43 sounds good 16:39:47 brain461_: very good 16:39:57 what about specs in git publicly? 16:40:09 public gerrit I mean with specs? 16:40:20 where would the one get specs now to build packages? 16:40:31 is there somewhere discussing what this is needed for? 16:40:38 That's side project of building openstack packages, but I'm going to add documentation for it though 16:41:06 xarses: it's about building packages without OBS or something 16:41:13 part of deploy from master efforts 16:41:17 Ok, thanks 16:41:22 xarses: at the moment, we build rpm packages for fuel only 16:41:29 and finally to have ability to deploy vanilla openstack, not MOS 16:41:39 is there an overview of the whole deploy from master effort that could provide context for side projects like this one? 16:41:49 brain461_: what do you mean for fuel only? 16:41:59 does it include openstack packgaes? 16:42:20 he means fuel-main only builds rpms of fuel components 16:42:21 I mean, during an iso build. It will include both fuel and openstack packages 16:42:33 what about os packages? 16:42:43 like ovs, qemu etc? 16:43:04 it will or it already does for RPM for openstack components? 16:43:08 should these packages be built during ISO compiling? 16:43:23 brain461_: not necessarily 16:43:25 it already does rpm packages for OS 16:43:34 I might want to build a package, but then use it as I want 16:43:44 DEB packages for OS are planned for 6.0 16:44:04 ok. brain461_ , so do you have specs for this whole feature? 16:44:56 there is a BP and specs for building OS packages, going to add specs for fuel packages as well 16:45:21 brain461_: pls share a link to bp with everyone 16:45:42 https://blueprints.launchpad.net/fuel/+spec/openstack-from-master 16:45:45 #link https://blueprints.launchpad.net/fuel/+spec/openstack-from-master 16:46:19 that's not a spec 16:46:38 https://github.com/stackforge/fuel-specs/blob/master/specs/5.1/openstack-from-master.rst 16:46:38 I'm not really happy to see it merged https://review.openstack.org/#/c/101868/ 16:46:40 there is a link to the spec 16:46:47 with just a review from dpyzhov only 16:47:04 should I limit merge to fuel-specs to myself only to avoid this.. ? 16:47:14 there's a link to something that looks like an unfilled template of a spec 16:47:38 angdraug: you are right, lots of gaps 16:47:47 ok let's get it sorted out. brain461_ to send all links we have to mailing list 16:47:53 we might need to reopen the effort 16:48:00 and get enough people to review 16:48:22 otherwise we might end up with reimplementation of the whole thing later, thus wasting resources 16:48:47 looks like we've discussed next topic as well 16:48:57 moving on? 16:49:08 brain461_: thanks for keeping pushing this even with my complaints - this work is really needed for us all 16:49:38 #topic how doc team should work with bugs (dmitryme) 16:49:38 with the move to external gerrit we are not only making real community story, but coming with improving our dev workflow 16:50:02 yep, seems like its my turn 16:50:05 so, basically the problem is doc team (Meg and Irina) are working on documenting known issues. They need some way to distribute workload among themselves. They want to do that by assigning bugs to each other. But at the same time, bugs are already assigned to engineers working on them. 16:50:12 I propose the following approach to solve the problem: we create another series 'doc' in addition to '5.0.x', '5.1.x', etc., we have right now in launchpad. Then whenever one of the doc team wants to assign a bug to herself, she adds 'doc' series as affected and assigns the bug to herself within that series. In launchpad bugs having several series affected can have different assignee for them. 16:50:26 I strongly object 16:50:29 I against of creating another series 16:50:36 we shouldn't overload release series with this new meaning 16:50:37 let's use tags for it 16:50:38 wow 16:50:52 you guys are like mind-connected 16:50:57 :-) 16:51:01 ) 16:51:08 our docs team is not large enough for the workload distrubution to become a problem 16:51:13 separate mirror bugs are not good to 16:51:20 too* 16:51:35 as long as engineers are still working on a bug, it's too early to document it 16:51:39 I think tags, and their periodical syncs should solve the issue 16:51:57 mihgen: +1 16:51:57 their == engineers 16:52:21 like if there is a tag "meg" , than it means meg took the bug for documentation 16:52:21 ok, as for me, one more series is not a big deal 16:52:24 would that work? 16:52:48 I don't think even that is really necessary 16:52:49 dmitryme: one more series is pretty heavy for LP 16:52:51 but if you guys don’t really like it, I will propose them to use tags for assignment 16:52:53 fuel-docs as a tag works fine 16:53:10 it basically needs to put all right statuses, milestone, too many clicks 16:53:35 fuel-docs will not work. They need to separate work between them 16:53:47 there's only two of them, how hard can it be? 16:53:49 like what is doing tech writer #1, and what #2 16:54:03 what happens if we have 10 tech writers? 10 tags? 16:54:04 yep, like mihgen said, the will need tags like doc-irina and doc-meg 16:54:06 they might start documenting same bug - dups without knowing it 16:54:24 10 tags might be fine too, why not? 16:54:25 they have each other's emails 16:54:40 and there's etherpad and google sheets 16:54:50 emails and what? Nope that won't work 16:54:54 let me simple exampel 16:54:57 remember, we're only talking about 'known issues' category of bugs 16:54:59 they should take ownership of the bug like th engs do 16:54:59 and still we havae only too. DOn’t overengineer here ;) 16:55:02 new bug apperaed and fixed 16:55:06 xarses: +1 16:55:09 so how I gonna find it? 16:55:18 xarses: -1 16:55:32 because they start working at the same time as dev is in progress 16:55:34 if bug is high or critical and has docs tag on it 16:55:40 it shouldn't be fix committed until its documented 16:55:46 so it should be still on dev 16:55:50 it should be assigned to one of the tech writers 16:56:11 angdraug: it's gonna be impossible to calculate HCF or anything 16:56:19 angdraug: err, that is the first time I hear that 16:56:22 I won't be able to separate dev from docs 16:56:48 looks like we need email discussion in openstack-dev about it 16:56:49 why would you want to? 16:56:56 if docs are not ready, you can't release 16:56:59 LP doesn't allow to do everything so we are working around 16:57:05 angdraug: +1 16:57:13 3 minutes 16:57:21 dmitryme: can you post email to openstack-dev [Fuel] with this question? 16:57:23 yes, lets move this to email 16:57:29 angdraug: but it could be that an engineer and doc-writer work in parallel on the bug 16:57:37 exactly 16:57:37 mihgen: yep, will do 16:57:44 that's what I'm talking about 16:57:48 you can also have more than one engineer working on a bug 16:57:50 same situation 16:57:54 all right folks great meeting I think 16:58:10 thanx everyone 16:58:17 those who have action item to share specs - let's don't hesitate and post email to openstack-dev 16:58:25 not exactly, I think about assigned engineer as the one responsible 16:58:28 and provide short description on what you are working on 16:58:55 even if a team works on the bug 16:59:15 #topic announcement Fuel september meetup 16:59:18 #link https://etherpad.openstack.org/p/fuel-september-meetup-2014 16:59:34 folks we gonna run it in Silicon Valley 15-26 of sep 16:59:40 anyone is welcome to join 16:59:46 there not many of us but still 16:59:54 thanx mihgen 17:00:01 #endmeeting