17:00:50 #startmeeting ironic 17:00:51 Meeting started Mon Jul 25 17:00:50 2016 UTC and is due to finish in 60 minutes. The chair is jroll. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:52 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:54 The meeting name has been set to 'ironic' 17:00:55 o/ 17:00:56 o/ 17:00:56 o/ 17:00:57 o/ 17:01:05 o/ 17:01:05 o/ 17:01:06 hello 17:01:29 o/ 17:01:42 o/ 17:01:45 hey everyone 17:01:51 * jroll gets started 17:01:55 \o 17:01:57 #topic Announcements and reminders 17:02:13 o/ 17:02:16 #info ironic now has the supports-upgrade and follows-standard-deprecation tags \o/ 17:02:38 woot! 17:02:46 o/ 17:03:01 Many thanks to everyone who's worked on upgrade testing! 17:03:12 #info some discussion on nova/ironic interactions at nova's midcycle last week, please do read: 17:03:14 #link http://lists.openstack.org/pipermail/openstack-dev/2016-July/099922.html 17:03:28 oh great news from the mid cycle jroll 17:03:40 o/ 17:03:59 any other announcements or reminders? 17:04:25 OneView CI has been down due to power outage problems that affected underlying infra 17:04:40 fixing it today, should be running soon 17:05:02 cool 17:05:07 iLO CI is also down due to infra getting shifted 17:05:22 should be up shortly 17:05:27 I noticed dell CI is failing very quickly as well 17:05:38 like, 7 second runs 17:05:46 I assume that isn't an ironic issue :) 17:06:35 ok, moving on 17:06:41 #topic subteam status reports 17:06:51 those are here 17:06:53 #link https://etherpad.openstack.org/p/IronicWhiteBoard 17:06:56 line 81 or so 17:07:31 I'll give folks a few to update / look over those 17:08:03 * natorious waves o/ 17:08:22 o/ 17:09:00 jroll: wrt the network isolation/nova patch, what does 'may still be able to..' mean? If the patch is reviewed and is good to merge? Or something else? 17:09:38 rloo: we missed the non-priority feature freeze, people seem supportive but haven't got confirmation from PTL 17:09:59 jroll: ok, so we should review it asap then. 17:10:11 jroll: also, is that the only outstanding patch (w/o portgroups?) 17:10:14 rloo: indeed 17:10:16 and yes 17:10:53 jroll: gotcha, thx. 17:11:40 looks like a lot of progress this week \o/ 17:12:28 jroll: The Dell hardware is being changed, so builds are broken. We are looking into it. 17:12:34 rpioso: noted, thanks 17:13:53 okay, any questions/comments on the subteam updates? 17:14:49 #topic Alternatives for the proposed approach at https://review.openstack.org/#/c/331564/11/specs/approved/boot-params.rst 17:14:54 #link https://review.openstack.org/#/c/331564/11/specs/approved/boot-params.rst 17:14:55 that's me :-) 17:14:56 lucasagomes: you're up 17:14:58 :D 17:15:03 so, looking for altenatives/suggestions here 17:15:12 The tl;dr of what the spec is proposing is: Having a way to the users to pass custom kernel command lines when configuring the bootloader for local boot 17:15:39 and current it proposes to use the standard way to do it with grub, which is editing the /etc/default/grub and then regenerating the grub.conf 17:15:51 the thing is that, in Ironic, we consider touching the files in the user image as a layer violation 17:16:14 and the alternatives that I can think of sucks 17:16:16 is there any reason this can't be done with port-boot automation and a reboot? 17:16:35 that's one alternative, have cloud-init to set it and reboot the node 17:16:35 (and maybe have ironic do the initial grub.conf to skip the first reboot) 17:16:45 cloud-init, or ansible, chef, etc 17:17:01 yeah, the only thing is that we could avoid that if it was configured by Ironic 17:17:05 no extra reboot, which saves time 17:17:15 and also is a bit weird to have a node to reboot twice after active 17:17:26 if ironic did the initial config you could skip the extra reboot 17:17:32 yes 17:17:43 just a bit of a duplication thing 17:18:02 another alternative is to use "grubby" which is a command line tool that can add/remove kernel parameters from the grub bootloader 17:18:08 but ubuntu doesn't package it :-/ 17:18:12 I kinda felt that it wasn't a layer violation as we are installing the boot loader anyway, so I looked at it like just some configuration of that 17:18:28 NobodyCam, yeah, that's kinda way I wanted to bring it here 17:18:37 I know it does touches the files in the image, but, that's the standard interface 17:18:43 in some cases, we're installing the bootloader (partition image) 17:18:52 but not in all cases 17:18:56 what about images that don't use grub? 17:19:01 right, but in no cases are we touching files in the rootfs 17:19:08 right 17:19:11 which seems to be the primary objection 17:19:12 devananda, exactly, that's what this spec is proposing. For when installing the bootloader for a partition image 17:19:15 which I tend to agree with 17:19:19 That's the first of many 17:19:26 my second objection would be in line with natorious's comment 17:19:31 sure 17:19:38 essentially we'd be adding an Ironic feature that is tightly coupled into the image 17:19:39 natorious, currently we only support grub instalation as the bootloader for patition image 17:19:43 which seems like a pretty bad idea 17:19:50 natorious, for full disk image you can use whatever 17:19:51 it might be a good time to start full in on efi 17:20:03 which would open up options 17:20:12 Another question I have is, why is this an Ironic-specific problem/ 17:20:15 bc that would be its own partition 17:20:26 Why shouldn't this be solved via nova, using more standard ways of configuring instances/ 17:20:34 JayF: good question, what does nova do here? do they do it? 17:20:43 * lucasagomes don't know 17:20:54 jroll: I don't think they support anyhting like this. 17:20:57 nova doesn't need efi boot w/ legacy emulation 17:21:14 and pygrub 17:21:24 natorious: I mean, how does a user give kernel args to nova, and what is done with them? 17:21:32 jroll: which is kinda my point; if nova supported it and we needed to implement it, I'd be slightly less weirded out by it, but as it sits, it seems like we're implementing a feature that has a small scope of usefulness while extending ironic's scope considerably 17:21:52 lucasagomes: also, who is meant to put the data in instance_info? operator? nova? 17:22:07 even if set in nova, it has to be updated to grub via another service or agent by reading it.. 17:22:14 skramaja, ^ is the author here 17:22:21 oh 17:22:34 and then reboot.. 17:22:37 skramaja: I understand that, I'm trying to figure out how this works as a whole 17:22:53 but there are services which are dependent certain args.. 17:23:04 like DPDK depnds on iommu 17:23:09 I didn't think nova passes in kargs outside of pygrub 17:23:11 ah, I see, the operator sets the instance_info 17:23:18 before enbling DPDK, immou has to be enabled.. 17:23:25 as someone that has needed to do this before (but with nova and VMs) we jsut edited our iamges and provided them to nova and it worked fine 17:23:30 jroll: nova sets instance_info, not operator ... 17:23:32 which... doesn't seem terribly useful to the user? 17:23:36 clarkb: that's kinda what I'm feeling the proper solution is 17:23:37 devananda: line 147 17:23:44 which worked fine for us eg nova/ironic don't need to know or care 17:23:46 clarkb: which is why I'm pushing back against ironic fiddling in etc 17:23:48 clarkb: ++ 17:23:55 clarkb: yep, makes sense 17:24:00 clarkb: JayF modifying image is not scalable.. 17:24:12 skramaja: building your own image is very scalable 17:24:31 because in case differential clusters, u need to build multiple images.. 17:24:34 (we set kernel boot params to limit available system memory fwiw) 17:24:59 we can use introspection rules via ironic to set specific args for isolcpus.. 17:25:03 skramaja: yea, build one image per hardware type, with what ever kernel params you need baked into it 17:25:47 or build local tooling to edit an existing image 17:26:00 * lucasagomes would prefer that ^ over creating loads of images 17:26:30 devananda: it is not just one set of parameters.. ther are multiple criterias we need to multiple combinations.. cpu isolatio, hugepages, iommo 17:26:49 lucasagomes: I think it is something that has been done in the community several times too :) 17:27:10 TheJulia, probably more people have hit that yeah 17:27:18 It sounds like this might be a larger issue though 17:27:25 given clarkb pointed out they had it with nova and vms 17:27:26 skramaja: and you can't change these from within the instance after its been deployed -- and just reboot it? 17:27:44 which gives me even more pause to Ironic doing it w/o Nova cooporation or at least some consensus on the ML 17:28:00 devananda: yes.. it is not possible because certain drivers loading like DPDK depends on immou flags 17:28:05 and hugepages enabling.. 17:28:48 right, my primary objections are: 1) reaching into the instance and touching /etc, and 2) that there's no interface to the user here, only the operator can set it 17:29:07 if we can solve those, I'm okay with it 17:29:14 otherwise I'd prefer to leave this to post-boot automation 17:29:27 jroll: what about 3) thinking about how something like this fits in the larger ecosystem 17:29:42 from clarkb's comment, it's clear this isn't unique to ironic, so I'm loathe for us to move on it w/o a larger consensus 17:29:43 jroll: if only the operator can set it, this proposal doesn't make any sense to me 17:29:47 JayF: that ties into (2) imo 17:29:59 jroll, 1) we can add a dependency on "grubby" so we don't touch files directly, but may not work on all distros 17:30:09 devananda: indeed, except for deployments where op==user 17:30:12 if the operator is setting kernel boot flags persistently on a Node, that has no bearing ... 17:30:13 lucasagomes: we're still modifying, even if we use an interface to it 17:30:14 yea, well... 17:30:28 lucasagomes: it wouldn't change the nature of my objection 17:30:30 JayF, but we do modify it when we install the bootloader 17:30:39 lucasagomes: not the root fs, only the /boot, correct? 17:30:42 jroll: I agree with your objections 17:30:42 we currently mount the partition and run grub-install 17:30:46 also, it isn't clear from this spec if this is only for partition images or also whole disk images 17:30:55 JayF, right and grubby does not modify /etc either 17:31:00 only grub.conf afaik 17:31:14 lucasagomes: if it doesn't change etc/default/grub whatever it does would be non-persistent 17:31:19 lucasagomes: and wouldn't really fix the issue 17:31:21 the downside of it is that, it's not persistent on a kernel update 17:31:55 JayF, right, that's why I added it to the meeting, cause the alternatives to changing default/grub are not very ideal 17:32:02 I also agree with almost every objection raised thus far. I feel like reaching into the image is a bad idea... if we want to provide a modular interface to do_something(), thats different and can be site/deployment specific... and truthfully that is what this feels like, pushing a environmental requirement into the deployment process 17:32:32 TheJulia: "pushing a environmental requirement into the deployment process" <-- A+++ summary 17:32:42 TheJulia: agreed 17:32:52 JayF: I've started and stopped typing that like 10 times now, so I finally just typed it 17:32:56 ok so post-deploy scripts 17:33:11 skramaja, even if not ideal, is it viable ? 17:33:26 and furthermore, there already exist ways to achieve the desired goal (bake the needed parameters into an image) 17:33:27 lucasagomes: i have to check 17:33:35 lucasagomes: I specifically suggested in IRC, that getting something in the configdrive and using an agent to set the options was an alternative I could be on board with. Then Ironic sets the opts for firstboot, and the agent persists them 17:33:49 lucasagomes: but that would require coming through nova and doing the work there first, since they own/manage configdrives 17:33:49 which we support and maintain the boundary between deployment process and tenant workload 17:33:56 JayF: which would depend on nova having an interface for this :) 17:33:59 yeah 17:34:11 lucasagomes: (although admittedly telling someone to go to nova first is slightly more helpful than telling them to screw off, today, lol) 17:34:21 :D 17:34:40 yeah, def we should see in nova how an interface for it would look like in parallel 17:34:56 * jroll wishes folks wouldn't complain about other openstack project teams here :/ 17:34:58 ok, so we can proceed from here, thanks for the ideas/suggestions/comments all 17:35:14 thank you for bringing it up lucasagomes 17:35:20 yeah, thanks lucasagomes :) 17:36:09 okay, next topic 17:36:15 there's a couple RFE reviews, should be quick 17:36:27 #topic RFE: Validate iLO SSL certificates in the iLO driver 17:36:44 I wish this was called "use custom CA certs to validate iLO SSL certs", but oh well 17:37:03 #link https://bugs.launchpad.net/ironic/+bug/1599710 17:37:03 Launchpad bug 1599710 in Ironic "[RFE] Validate iLO SSL certificates in the iLO driver" [Wishlist,In progress] - Assigned to Shivanand Tendulker (stendulker) 17:37:11 this adds a ca_cert option for ilo communication 17:37:18 jroll: I can change it 17:37:20 makes sense to me, any objections? 17:37:27 stendulker: I'll comment on the review :) 17:37:37 jroll: thank you :) 17:37:51 no objections here 17:37:52 nope 17:38:17 jroll: not a complaint, just they're more busy than we are :) 17:38:23 specs cores? JayF devananda rloo lucasagomes NobodyCam ? 17:38:38 no objections either 17:38:44 none here 17:38:47 i'm fine 17:38:52 cool, approving 17:39:10 and the second RFE on the agenda was approved earlier today 17:39:10 I have +2s on that spec already 17:39:12 woo 17:39:25 jroll: yeah, sorry, i forgot to delete that from the agenda 17:39:26 Wait, a question then 17:39:34 I thought you weren't supposed to approve an RFE that had a spec 17:39:36 until the spec landed 17:39:40 there's a spec for this? 17:39:42 but this spec is blocked because the RFE isn't approved? 17:39:48 it's a code patch 17:39:50 not a spec 17:39:56 afaict 17:40:00 lol, I'm not used to being core /o\ 17:40:01 https://review.openstack.org/#/c/338791 17:40:02 * thiagop remembers deadlocks 17:40:03 lol 17:40:05 saw my +2 and presumed spec 17:40:22 #link https://review.openstack.org/#/c/338791/ 17:40:24 :D 17:40:26 it's code 17:40:36 stendulker: posted comments, if you update that I'm happy with it 17:40:46 okay, let's move on 17:40:47 jroll: Will update it 17:40:49 #topic open discussion 17:40:51 stendulker_: thanks :) 17:41:08 anyone have anything for open discussion? 17:41:21 jroll: there is a config drive related RFE for quiet some time https://review.openstack.org/#/c/338791 17:41:43 stendulker_: wrong link? 17:41:46 stendulker_: that's a wrong link, I think? That's what we just looked at 17:41:47 jroll: Would you please comment on fate of that as well 17:42:02 jroll: https://launchpad.net/bugs/1599710 17:42:02 Launchpad bug 1599710 in Ironic "[RFE] Validate iLO SSL certificates in the iLO driver" [Wishlist,In progress] - Assigned to Shivanand Tendulker (stendulker) 17:42:08 The dynamic allocation for OneView drivers code was refactored to ease the life of you guys reviewing it. If you'd be so kind as to take a look, it is much appreciated: https://review.openstack.org/#/c/286192/ 17:42:11 stendulker_: also wrong link? 17:42:16 stendulker_: that's literally what we just talked about and approved? 17:42:31 jroll: https://bugs.launchpad.net/ironic/+bug/1493328 17:42:31 Launchpad bug 1493328 in Ironic "No Config drive support for whole disk images deployed using iscsi based deploy" [Medium,In progress] - Assigned to Shivanand Tendulker (stendulker) 17:42:47 btw, TheJulia: Thanks for noting that. I'm looking into it yet today. 17:43:00 thiagop: no problem :) 17:43:01 Corresponding review https://review.openstack.org/#/c/230924/ 17:43:03 stendulker_: really?! O_o 17:43:34 wow, how did we miss that 17:43:34 jroll: review raised since Oct'15 17:44:19 stendulker_: I'll take a look at that patch today 17:44:20 stendulker_: sounds like a bug to me, no need for RFE approval 17:44:22 but 17:44:30 would cores help push that through please? 17:44:33 that's a big miss :( 17:44:40 it touches all ironic-lib, ironic and ipa 17:44:47 yeah 17:44:47 all three patches in review 17:44:56 stendulker_: can you link the other two in the bug? 17:45:15 oh - ouch 17:45:18 jroll: they are 17:45:31 jroll: same bug id being used 17:45:38 stendulker_: I only see two, 230924 and 225115 17:45:54 it looks like last update to the ironic patch was 6 weeks ago? 17:46:28 jroll: ipa is not linked. Will do that https://review.openstack.org/#/c/296466/ 17:46:31 yeah, that could use an update 17:46:34 stendulker_: thanks 17:46:51 stendulker_: if you can update these, I'll add them to my review queue 17:46:52 ironic lib is a dependency 17:46:53 stendulker_: so if you can get the ironic and IPA patches updated, I'd love to get this done soon 17:46:56 ah right 17:47:07 ironic patch cannot merge without ironic lib 17:47:07 okay let's push the ironic-lib patch through, and I'll release it 17:47:08 I'll look over that ironic-lib patch today 17:47:25 this aligns with our desire to get those shell scripts outta ipa too 17:47:27 ironic lib-> ironic -> ipa 17:47:28 it's a win all the way around 17:47:33 is the dependency list 17:47:43 also, is it worth setting up gate job for this configuration, perhaps limited to ironic-lib or IPA patches? 17:48:16 devananda: I think someone is working on whole disk image + iscsi CI 17:48:23 and configdrive is always enabled in CI 17:48:26 cool 17:48:56 Also any help on thsi short spec would be useful https://review.openstack.org/#/c/230274/ 17:49:11 https://launchpad.net/bugs/1526755 17:49:11 Launchpad bug 1526755 in Ironic "[RFE] Baremetal provisioning using UEFI secure boot mode" [Wishlist,In progress] - Assigned to Shivanand Tendulker (stendulker) 17:49:37 Few cores have already reviewed it. 17:50:02 stendulker_: are there plans for other drivers to support uefi-secure-boot ? 17:50:24 devananda: do not know 17:50:26 * devananda looks towards the other driver maintainers 17:51:12 devananda: not in short term for OneView 17:52:05 hm, seems like nova shouldn't have anything to do with the boot mode for the deploy ramdisk 17:52:26 I see others are okay with this thuogh, so someone correct me if there's a reason for that 17:52:34 secure boot enabled deploy ramdisk support is already present in DIB 17:52:51 stendulker_: the spec seems to indicate that secure boot is a property of the Nova flavor 17:53:13 stendulker_: but doesn't this also depend on the user image? 17:53:19 devananda: yes. That is already there when iLO drivers enabled secure boot 17:53:23 devananda: this is about the deploy ramdisk 17:53:36 ah. gotcha 17:53:38 devananda: yes. user image also needs to be secure boot enabled 17:53:46 which makes me wonder why nova cares :) 17:53:50 * jroll will take it to the spec 17:53:55 devananda: partition image creation support is available in DIB 17:54:11 devananda: whole disk image creation patches under review in DIB 17:54:31 jroll: thank you 17:55:17 any other topics? 5 minutes left 17:55:21 just a quick request for reviews on https://review.openstack.org/#/c/272658 17:55:23 :;p 17:55:37 I'll comment on the spec too. I'm not sure why this is exposed to users orthogonally to the security of the instance image 17:56:28 devananda: thank you for the review 17:58:37 Great meeting Thank you all :) 17:58:54 yep, thanks all 17:58:56 #endmeeting