19:00:11 #startmeeting Ironic 19:00:11 #chair devananda 19:00:11 Meeting started Mon Feb 24 19:00:11 2014 UTC and is due to finish in 60 minutes. The chair is NobodyCam. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:14 Welcome everyone to the Ironic meeting. 19:00:16 The meeting name has been set to 'ironic' 19:00:18 Current chairs: NobodyCam devananda 19:00:19 hi all! 19:00:24 Ahoy! o/ 19:00:27 Who's here for hte meeting 19:00:27 o/ 19:00:31 me 19:00:31 o/ 19:00:33 moi 19:00:42 o/ 19:00:49 Of course the agenda can be found at: 19:00:51 #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting\ 19:01:00 #topic Greetings, roll-call and announcements 19:01:07 Welcome all! 19:01:38 announcments: 19:01:48 Nova driver is currently blocked 19:02:30 o/ 19:03:18 Hi 19:03:21 we Need to have everything in place for us to graduate, While we are blocked this does not mean we can not graduate 19:03:40 just letting us know we are currently not ready to 19:03:53 The nova team has pushed the deprecate-baremetal-driver BP out to Juno, and said that they won't even start reviewing the driver code until we have functional & integration testing with devstack & tempest 19:03:55 can someone list out the blockers? I can help out 19:04:35 k4n0, I think it's this patch: https://review.openstack.org/#/c/70348 19:04:46 k4n0: at a high level, we're blocked on devstack support 19:04:51 ok 19:04:58 #link #topic Greetings, roll-call and announcements 19:05:00 ifarkas: yep, that's one of them 19:05:01 gah 19:05:11 #link https://review.openstack.org/#/c/70348 19:05:37 once we have devstack into a state where it can prepare the environment 19:05:52 (create a net bridge, some VMs, enroll them in ironic, etc) 19:06:02 we have had some really good progress over this last weekend! 19:06:33 max_lobur: and at least one other have gotten neutron work! 19:06:36 then I should be able to create an infra job that runs tempest against devstack with nova.virt.ironic.IronicDriver instead of nova.virt.libvirt.LibvirtDriver 19:06:56 and run that as an experimental job against the Nova driver patch series ... 19:06:59 #link https://review.openstack.org/#/c/71026/ 19:07:14 :) 19:07:24 agordeev said in channel this morning taht he had solved the networking issue we've been running into 19:07:44 max_lobur: did I read you also got it working? 19:07:44 w00t! 19:08:04 I haven't seen the changes that make this happen yet, but will test agordeev's patch later today 19:08:05 yep, I helped to overcome neutron 19:08:14 max_lobur: can you summarize what was fixed? 19:08:53 vms rolled by devstack now should be able to get IP from neutron dhcp server 19:09:02 ahh! great 19:09:04 this was our main problem 19:09:07 yep 19:09:38 max_lobur: where was the hart of the bug that stopped us b4? 19:09:48 PXE driver was correctly settingthe dhcp boot options last week, but in my tests VMs weren't getting IPs. so yea, taht sounds like you guys got it 19:10:09 there were a few problems 19:10:21 1st - we did not tagged our traffic 19:10:38 2nd we did not registered ports for our vms in neutron 19:10:54 * I mean VLAN tags 19:11:31 seems that's all :) 19:11:43 ahh will we be seeing a patch for that soon? 19:11:52 and great work !!!! 19:12:10 I thought agordeev already included this, though I did not look yet 19:12:25 and did not test within devstack 19:12:25 #link https://review.openstack.org/#/c/70348/9/lib/ironic 19:12:27 looks like it's there 19:12:38 perfect :) w00t 19:12:40 though there are a lot of hard-coded IPs and #FIXEM's :( 19:12:48 lol 19:12:57 we can fix that 19:13:04 s/a lot/some/ 19:13:10 yea, let's iterate on taht today 19:13:17 when i was testing it, I added the ports and all, but the a neutron port-show command showed that port status as DOWN :( 19:13:21 this leads us to: 19:13:31 it was before the changes today 19:13:32 #topic I-3 Planning 19:13:54 another thing blocking us -- just the length of the review queue itself. there are some large'ish patches up there that we need to merge soon 19:14:21 lucasagomes: i haven't figured out why the ports are DOWN either yet 19:14:28 lucasagomes: hmm, I did not even look to the status, I think it's ok with any status. Also I remember some issues in neutron led to incorrect status 19:14:49 but this was in grizzly 19:14:49 right, yeah I will give it a go again tomorrow with the new changes 19:14:57 maybe they still there 19:15:00 so, I3 milestone will be tagged next week 19:15:12 devananda: lucasagomes does the nova driver need to turn up the Neutron port when it attaches it to the nodes port 19:15:29 we should be aiming to land any remaining major features this week -- eg, console support, pxe driver timeouts, etc 19:16:01 I'd like to land the NodeManager refactoring as well, if folks feel that it's ready 19:16:15 + 19:16:33 NobodyCam, I think not the nova driver, it should be part of the _update_neutron() call in Ironic I bet 19:16:33 we missed our review jam this morning... maybe tomorrow morning? 19:16:33 need to rebase my threading patch for vendor passthru 19:16:42 I'll do these days 19:16:47 I will toss up a quick patch to finish https://blueprints.launchpad.net/ironic/+spec/keep-powered-off-nodes-off today 19:16:53 NobodyCam: +1 19:17:19 there's already sync_power_state, but instead of just updating the DB, it needs to respect waht's in the DB 19:17:19 lucasagomes: ya that sounds better to me 19:17:33 and I think we can mark https://blueprints.launchpad.net/ironic/+spec/add-neutron-support as completed now 19:17:48 since many of us have confirmed that neutron dhcp-extra-opts are being set 19:17:52 yes? 19:17:53 I'm not sure about tomorrow, but will update review doc with tested patches as match as possible 19:18:16 *as much 19:18:20 devananda: with agordeev's patch? or with out 19:18:29 NobodyCam: I'm up for a review jam tmw morning, fwiw 19:18:46 NobodyCam: with agordeev's devstack patch -- but taht's just setting up the ENV 19:18:53 NobodyCam, yeah review jam tomorrow seems fine for me as well 19:19:12 NobodyCam: I meant the the add-neutron-support BP for Ironic is complete 19:19:24 devananda, before marking it as complete, let's try the devstack patch first 19:19:31 devananda: ahh ack +1 19:20:00 lucasagomes: neutron is getting setup 19:20:08 lucasagomes: what i mean is, ironic is correctly configuring neutron's dhcp opts, if neutron is present 19:20:24 I think we can call it done as this is a env on / off issue 19:20:26 we probably need to improve handling around errors, etc 19:20:51 devananda: +1 in lots of places 19:20:52 and refactor it so that PXE driver doesn't depend on neutron (eg, add a thin abstraction and allow it to work with other DHCP services) 19:21:24 devananda: for graduation or in the J cycle? 19:21:33 ^^ for refactor 19:22:03 NobodyCam: J 19:22:15 then +1 19:22:21 :-p 19:22:49 ook so that brings me to : 19:23:01 so we also still have a lot of HIGH bugs marked as Needs Review and targetd to I3 19:23:04 #link https://launchpad.net/ironic/+milestone/icehouse-3 19:23:13 I'd like to see us focus on those too :) 19:23:24 (i know, focus in a dozen places is not really focusing..) 19:23:35 +1 19:23:49 devananda: how about the first 15 - 30 minutes for review jam is also used for bug jamming 19:23:52 :-p 19:24:01 NobodyCam: ++ 19:24:06 ++ 19:24:12 I would like to give an update on iLO driver https://review.openstack.org/#/c/73787/. We submitted a new patch set today, please review and provide comments. Thanks! 19:24:16 https://bugs.launchpad.net/ironic/+bug/1279206 look already fixed for example 19:24:26 NobodyCam, +1 19:24:55 Hi wanyen, can you hold that for open Discussion 19:25:02 sure. 19:25:16 #toipc code cleanup 19:25:32 "with mock" vs "@mock" & "node['property']" vs "node.property" 19:25:33 NobodyCam: typo :p 19:25:42 #topic code cleanup 19:25:46 :) 19:25:46 #topic code cleanup 19:25:49 lol 19:26:09 lucasagomes: was the mock vs @mock yours 19:26:18 right, so there are some large patches doing code cleanup, and some large patches making functional changes 19:26:24 "with mock" vs "@mock" & "node['property']" vs "node.property" - vote up! 19:26:33 was? not really, I used @mock in the nova ironic volume driver tho 19:26:37 it's useful and cleaner 19:26:51 I don't see there's a mock vs @mock here 19:27:08 if we want to mock an attribute of a class we still need to use "with mock" 19:27:26 or change mocks inside the test 19:27:29 I agree, @mock is pretty clean, and the patch changing it looks reasonably small 19:27:51 I like the idea of both of these thing, standardizing code and clean up.. but we are up agenst the wall ATM and I feel these are better addressed post I release 19:28:11 yea... and I didn't follow up from last meeting. sorry about that, guys 19:28:14 NobodyCam++ 19:28:30 i'll do that right after this meeting 19:28:39 NobodyCam, +1 19:28:46 NobodyCam, +1 19:28:58 NobodyCam: want to action me? 19:29:29 +1 for any way to cut the review queue :) 19:29:34 devananda: same action you where going to -2 some reviews to clean the queue up 19:29:43 yep 19:30:11 I think we all agreed thats a good thing at this point 19:31:07 #action devananda to -2 reviews that should be held until after IceHouse is cut 19:31:08 about the node['property'] vs node.property, I suggest we to follow markmc's comments at https://review.openstack.org/#/c/64336/ (Patchset #13) 19:31:18 too many windows 19:31:40 romcheg, max_lobur, NobodyCam, lucasagomes - what about vendor drivers? any of you at this point have time to work on those? 19:32:36 I have started looking at the Ilo driver but not the SeaMicro yet 19:32:36 lucasagomes: yes - I agree that we need to follow markmc's comment there as far as the object base class goes 19:33:07 #topic vendor drivers 19:33:15 devananda: I think ilo driver would be easier to review if it was splited onto smaller patches 19:33:22 but I also started looking on it 19:33:35 devananda: you mean to review them? 19:33:39 devananda, I think we should review the vendor drivers, because although we don't need it for graduation they are using our vendor_passthru interface and finding the gaps within it 19:33:40 I left the same comment inline 19:33:46 which is good for a design point of view 19:33:52 I can spend some time if there are no other urgent things 19:34:00 and will definitely review them this weekend 19:34:01 like the MultipleVendorInterface 19:34:03 max_lobur: yea. at least to provide feedback to the developers and, as lucasagomes points out, see what they are needing from the common code 19:34:10 devananda: ++ 19:34:13 https://review.openstack.org/#/c/70719/ and https://review.openstack.org/#/c/70720/ :) 19:34:26 devananda: ++ 19:34:42 lucasagomes: good point 19:34:57 I would be useful to test them on real hardware 19:35:05 iLO driver https://review.openstack.org/#/c/73787/ 19:35:18 #link https://review.openstack.org/#/c/73787 19:35:24 FWIW, i think it's important for us to provide some feedback to vendors on these drivers before the Icehouse release, even if it's not a release-critical item, and even if we don't merge them yet 19:35:31 wanyen, k4n0 thanks! I will take a look as soon as I can 19:35:35 #link https://review.openstack.org/#/c/70719/ 19:35:39 #link https://review.openstack.org/#/c/70720/ 19:35:53 as far as testing on real hardware goes -- I sent an email to the -dev list a while back 19:36:07 outlining what I think we should expect from Vendors that want to provide a driver 19:36:08 well, whether they work or not on the real hardware - I think it's vendor's responsibility hehehe :) 19:36:09 wanyen: is 73790 still needed or are you working with it 19:36:30 max_lobur: IMO, it's vendor's responsibility to prove that it works on the hardware they claim it works on 19:36:40 true 19:37:03 devananda: DO we require smoke test to show its working 19:37:05 max_lobur: and continue to demonstrate that via jenkins test runs triggered by any change to ironic code 19:37:18 Well while we expect vendors to be responsible we should also check everything we accept 19:37:32 NobodyCam: I proposed that we needed that by Juno release ... lemme find the email 19:37:50 romcheg: I don't disagree, but I'm not sure it's always possible 19:37:54 devananda: yea, but that will be possible once we have agreed the approach - e.g. vendor labs etc 19:38:12 i.e., with SeaMicro -- most of us probably don't have one of their machines lying around 19:38:15 I thought this discussion is planned for summit, right? 19:38:28 max_lobur: right 19:38:30 max_lobur: ++ 19:38:35 cool 19:38:36 My udnerstandign is that we changed scp to sftp so we don't need is https://review.openstack.org/#/c/73790/. But I will double check with the team. 19:38:36 Oh, nice. 19:38:46 SeaMicro is cool with Vendor labs in their premise 19:38:57 We can do third party testing for each ironic patch 19:39:05 will vendors provide a way to test their hardware right? cause I think it's important, we don't want to have broken code in trunk 19:39:17 #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/024823.html 19:39:25 lucasagomes: yes that is the plan see ^^^ 19:39:33 awesome 19:39:39 thank you devananda 19:39:54 k4n0: fantastic, glad to hear that :) 19:39:56 yea that's it 19:40:12 :) 19:40:19 k4n0: obviously, we need a way to test ironic in devstack/tempest upstream first -- that's the patch agordeev has been working on (linked earlier today) 19:40:36 devananda: Yes 19:40:46 k4n0: my goal would be for the same devstack/tempest job in your lab, just with a different Ironic configuration 19:41:04 devananda: Ideally only the driver info and confs will change :) 19:41:17 and instead of running tests for every patchset to related projects (eg nova/glance/etc) it would only run for ones in the ironic queue 19:41:31 k4n0: that how I Sea it (pun intended) 19:41:46 k4n0: some bits in devstack will change too - -instead of generating VMs to enroll with Ironic, you'll pass in a .csv (or something) with hardware info 19:42:01 k4n0: and maybe a bit of network difference, too 19:42:03 NobodyCam: I Sea micro changes required to agordeev patch to get them working for us 19:42:24 devananda: correct 19:42:46 k4n0: :-p 19:42:49 devananda: Since we have time till J-2 milestone, we can discuss this in detail after Summit? 19:42:57 k4n0: either at or after -- yes :) 19:43:11 devananda: Sure, at and after :) 19:43:33 devananda: I will get the CI setup done in our env till then 19:43:48 NobodyCam: I think that's it for the planned topics... anything else or shall we open the floor? 19:43:50 devananda: I am looking at Jay pipes blogs for setting up third party CI 19:43:50 wanyen: has your team looked to to what will be needed to test the Ilo driver? 19:44:21 one sec devananda 19:44:24 ack 19:44:26 You mean tempest and hardware? 19:44:36 k4n0: I'm available if you need assistance. 19:44:58 wanyen: yes so that the ILO driver will be tested with every commit 19:45:17 we can talk offline about that 19:45:21 yes. We looked into that and have done some tempest test. 19:45:24 jaypipes: Thanks Jay, will email you for sure 19:45:56 wanyen: we will need hardware to actually test the driver with 19:46:17 NobodyCam: no we won't :) 19:46:21 Hardware in our development lab, yes. 19:46:21 devananda: are we concerned about different versions 19:46:35 ie Ilo1,2,3,4 19:46:53 NobodyCam: the iLO team will need to have the automated smoke tests running in their own hardware env and reporting them back to gerrit 19:47:18 we will support ilo4 and beyond. We have tested on ProLiant gen7 & Gen 8 servers. 19:47:22 yes that is what I tried to say... 19:47:32 great 19:47:36 :) 19:47:46 ok then open the gates: 19:47:53 #topic Open Discussion 19:48:10 I want to take inputs about https://blueprints.launchpad.net/ironic/+spec/advanced-driver-api 19:48:18 * devananda looks 19:48:26 * NobodyCam clicks 19:48:33 ahh 19:48:35 that 19:48:50 vendor passthru ? 19:49:01 max_lobur: not exactly 19:49:04 max_lobur: No, we are exposing drivers via API 19:49:36 max_lobur: more like what rloo proposed here: https://review.openstack.org/#/c/73005/ 19:49:40 right, we have a /drivers now but it's pretty much list the current active drivers 19:49:46 we want to expand it? 19:49:50 Talking specifically about auto discovery using this API, would there be node registrations on discovery? 19:50:10 lucasagomes: Yes, we need to be able to call driver functions via API 19:50:27 lucasagomes: Without needing any node object 19:50:46 right 19:50:48 ahh 19:50:50 I see 19:50:58 Ahh! 19:50:58 this also raises the issue of request/response for long-lived requests 19:51:12 there are a few ways we could do that without a Node resource 19:51:34 * return Request-ID in the resposne header, let client poll for that 19:52:08 * let client pass a callback URI, and have server issue a request to client's API when done 19:52:16 devananda: Request-id looks good, but we should also have API to list all queued request-id's 19:52:23 k4n0: I see cases where operators may want auto enrollment but there is also a good case for not auto enrollong them. 19:52:36 NobodyCam: What is that case? 19:52:43 k4n0: security 19:52:56 request id looks simpler 19:52:57 k4n0: discover then compare results to part #'s 19:52:59 I think we need a conf switch for that type of option 19:53:06 and yes, that should be configurable 19:53:39 max_lobur: request-id is not that simple. it would require a separate scheduling service to queue up requests, track them, etc 19:53:54 a callback API is actually much simpler on the server side AND much more scalable 19:53:57 if we don't auto enroll them, what user should poll for? 19:54:07 max_lobur: or enroll-but-disable 19:54:19 ah, make sense 19:54:27 devananda: +1 for enroll-and-disable 19:54:30 k4n0: what do you think of a callback API for drivers? 19:54:31 enroll -> disable 19:55:09 eg, request would include a URL to which Ironic would POST any nodes it discovered 19:55:11 devananda: API user passes callback url for drivers to callback? 19:55:16 yes 19:55:27 devananda: cleaner than request-id 19:55:31 five minute bell 19:55:31 rather than the user polling Ironic to see when their request is complete 19:55:41 whcih also does not scale with large number of clients 19:55:48 and less heavy for the api than polling 19:55:54 devananda: correct, callback sounds good 19:55:55 a callback would allow Ironic to POST when it's done 19:56:07 devananda: that also seem a bit beter for ha stuff too? 19:56:12 yea 19:56:25 the other wrinkle is that there will be many conductor processes running 19:56:29 nodes are hashed across them 19:56:30 what if that callback failed? lost connection for a moment, etc 19:56:52 but drivers may run on 1 or more conductors (or all of them) 19:57:16 should a driver action be routed to one instance of that driver, or broadcast to all of them? 19:57:19 yeah callback sounds reasonable 19:57:21 max_lobur, retry? 19:57:47 retry discovering? and rolling up the nodes.. 19:57:50 devananda, maybe make it optional? 19:57:51 devananda: drivers can maintain state of current callback 19:57:53 looks long 19:57:53 ? 19:57:56 fan=[True|False] 19:58:03 broadcast* 19:58:19 lucasagomes: like unicast? 19:58:36 RPC layer supports fanout 19:58:42 NobodyCam, yeah, well some actions might need to be done on each condutor that contains that driver 19:58:44 others not 19:58:49 lucasagomes: right 19:59:01 so it might make sense to make broadcast optional 19:59:06 but vendor_passthru is not inspected within the API 19:59:22 one minte 19:59:23 discovery will need to be a separate URI outside of vendor_passthru 19:59:42 let's continue in ironic 19:59:44 *I think discovery will ... 19:59:47 devananda, yeah, we might even thing about putting vendor_passthru at / 19:59:51 can we move this back to -ironic? 19:59:56 and making node an parameter 19:59:56 sure :) 19:59:56 Sure 20:00:06 Thanks Everyone 20:00:07 thanks all! see you next seek! 20:00:10 *week 20:00:11 Thanks :) 20:00:11 thanks all 20:00:12 thanks 20:00:13 oh btw 20:00:13 :) 20:00:15 See ya! 20:00:15 thanks 20:00:16 thank you all we are moving back to our home channel 20:00:18 #endmeeting