| *** kumarmn has joined #openstack-meeting-alt | 00:03 | |
| *** kumarmn has quit IRC | 00:05 | |
| *** kumarmn has joined #openstack-meeting-alt | 00:05 | |
| *** superdan is now known as dansmith | 00:11 | |
| *** kumarmn has quit IRC | 00:15 | |
| *** hiro-kobayashi has joined #openstack-meeting-alt | 00:17 | |
| *** markvoelker has joined #openstack-meeting-alt | 00:20 | |
| *** markvoelker has quit IRC | 00:24 | |
| *** yangyapeng has quit IRC | 00:28 | |
| *** yangyapeng has joined #openstack-meeting-alt | 00:29 | |
| *** yangyapeng has quit IRC | 00:33 | |
| *** ijw has joined #openstack-meeting-alt | 00:36 | |
| *** ijw has quit IRC | 00:41 | |
| *** yangyapeng has joined #openstack-meeting-alt | 01:13 | |
| *** kumarmn has joined #openstack-meeting-alt | 01:15 | |
| *** hongbin has joined #openstack-meeting-alt | 01:17 | |
| *** yangyapeng has quit IRC | 01:21 | |
| *** yangyapeng has joined #openstack-meeting-alt | 01:21 | |
| *** kumarmn has quit IRC | 01:21 | |
| *** daidv has joined #openstack-meeting-alt | 01:27 | |
| *** caowei has joined #openstack-meeting-alt | 01:30 | |
| *** dalgaaf has quit IRC | 01:59 | |
| *** dalgaaf has joined #openstack-meeting-alt | 02:01 | |
| *** julim has quit IRC | 02:05 | |
| *** zhurong has joined #openstack-meeting-alt | 02:27 | |
| *** michchap has joined #openstack-meeting-alt | 02:41 | |
| *** markvoelker has joined #openstack-meeting-alt | 02:51 | |
| *** kumarmn has joined #openstack-meeting-alt | 03:10 | |
| *** kumarmn has quit IRC | 03:19 | |
| *** markvoelker has quit IRC | 03:25 | |
| *** valkyrie1 has quit IRC | 03:27 | |
| *** valkyrie1 has joined #openstack-meeting-alt | 03:36 | |
| *** valkyrie1 is now known as mguiney | 03:36 | |
| *** kumarmn has joined #openstack-meeting-alt | 03:45 | |
| *** sdake has quit IRC | 03:46 | |
| *** sdake has joined #openstack-meeting-alt | 03:47 | |
| *** sdake has quit IRC | 03:47 | |
| *** sdake has joined #openstack-meeting-alt | 03:47 | |
| *** julim has joined #openstack-meeting-alt | 03:50 | |
| *** julim_ has joined #openstack-meeting-alt | 03:52 | |
| *** julim has quit IRC | 03:54 | |
| *** yamamoto has joined #openstack-meeting-alt | 04:05 | |
| *** bhavik1 has joined #openstack-meeting-alt | 04:10 | |
| *** hiro-kobayashi has quit IRC | 04:10 | |
| *** bhavik1 has quit IRC | 04:16 | |
| *** ykarel|away has joined #openstack-meeting-alt | 04:24 | |
| *** caowei has quit IRC | 04:30 | |
| *** sankarshan has quit IRC | 04:36 | |
| *** sankarshan has joined #openstack-meeting-alt | 04:36 | |
| *** zhurong has quit IRC | 04:39 | |
| *** kumarmn has quit IRC | 04:42 | |
| *** hiro-kobayashi has joined #openstack-meeting-alt | 04:46 | |
| *** janki has joined #openstack-meeting-alt | 04:59 | |
| *** links has joined #openstack-meeting-alt | 05:11 | |
| *** markvoelker has joined #openstack-meeting-alt | 05:22 | |
| *** zhurong has joined #openstack-meeting-alt | 05:24 | |
| *** caowei has joined #openstack-meeting-alt | 05:27 | |
| *** anilvenkata has joined #openstack-meeting-alt | 05:31 | |
| *** gouthamr has joined #openstack-meeting-alt | 05:35 | |
| *** dsariel has quit IRC | 05:40 | |
| *** kumarmn has joined #openstack-meeting-alt | 05:43 | |
| *** hongbin has quit IRC | 05:43 | |
| *** kumarmn has quit IRC | 05:48 | |
| *** sridharg has joined #openstack-meeting-alt | 05:51 | |
| *** gouthamr has quit IRC | 05:54 | |
| *** yamamoto_ has joined #openstack-meeting-alt | 05:55 | |
| *** yamamoto_ has quit IRC | 05:55 | |
| *** juggler has joined #openstack-meeting-alt | 05:55 | |
| *** markvoelker has quit IRC | 05:55 | |
| *** yamamoto has quit IRC | 05:59 | |
| *** juggler has quit IRC | 06:03 | |
| *** janki is now known as janki|afk | 06:10 | |
| *** janki|afk has quit IRC | 06:10 | |
| *** janki has joined #openstack-meeting-alt | 06:11 | |
| *** janki is now known as janki|afk | 06:12 | |
| *** yamamoto has joined #openstack-meeting-alt | 06:16 | |
| *** julim has joined #openstack-meeting-alt | 06:17 | |
| *** julim_ has quit IRC | 06:17 | |
| *** shaohe_feng has quit IRC | 06:27 | |
| *** felipemonteiro_ has joined #openstack-meeting-alt | 06:37 | |
| *** absubram has joined #openstack-meeting-alt | 06:38 | |
| *** shaohe_feng has joined #openstack-meeting-alt | 06:40 | |
| *** hieulq has quit IRC | 06:41 | |
| *** kumarmn has joined #openstack-meeting-alt | 06:44 | |
| *** kumarmn has quit IRC | 06:49 | |
| *** felipemonteiro_ has quit IRC | 06:58 | |
| *** jbadiapa has joined #openstack-meeting-alt | 06:59 | |
| *** dtrainor has quit IRC | 06:59 | |
| *** marios has joined #openstack-meeting-alt | 07:09 | |
| *** marios has quit IRC | 07:14 | |
| *** julim has quit IRC | 07:15 | |
| *** marios has joined #openstack-meeting-alt | 07:19 | |
| *** armaan has quit IRC | 07:21 | |
| *** armaan has joined #openstack-meeting-alt | 07:21 | |
| *** alexchadin has joined #openstack-meeting-alt | 07:25 | |
| *** ykarel|away is now known as ykarel | 07:29 | |
| *** vds has joined #openstack-meeting-alt | 07:34 | |
| *** kumarmn has joined #openstack-meeting-alt | 07:45 | |
| *** janki|afk is now known as janki | 07:46 | |
| *** alexchadin has quit IRC | 07:49 | |
| *** kumarmn has quit IRC | 07:49 | |
| *** markvoelker has joined #openstack-meeting-alt | 07:53 | |
| *** matrohon has joined #openstack-meeting-alt | 07:58 | |
| *** rcernin has quit IRC | 07:59 | |
| *** armaan has quit IRC | 08:04 | |
| *** armaan has joined #openstack-meeting-alt | 08:04 | |
| *** alexchadin has joined #openstack-meeting-alt | 08:05 | |
| *** florianf has joined #openstack-meeting-alt | 08:15 | |
| *** zhurong has quit IRC | 08:18 | |
| *** zhurong has joined #openstack-meeting-alt | 08:21 | |
| *** ykarel is now known as ykarel|lunch | 08:22 | |
| *** hiro-kobayashi has quit IRC | 08:23 | |
| *** markvoelker has quit IRC | 08:26 | |
| *** julim has joined #openstack-meeting-alt | 08:40 | |
| *** oidgar has joined #openstack-meeting-alt | 08:44 | |
| *** kumarmn has joined #openstack-meeting-alt | 08:46 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** shu-mutou-AWAY has quit IRC | 08:50 | |
| *** kumarmn has quit IRC | 08:50 | |
| *** oidgar has quit IRC | 08:51 | |
| *** bauwser is now known as bauzas | 08:57 | |
| *** rossella_s has joined #openstack-meeting-alt | 09:01 | |
| *** alexchadin has quit IRC | 09:04 | |
| *** jtomasek has joined #openstack-meeting-alt | 09:05 | |
| *** alexchadin has joined #openstack-meeting-alt | 09:05 | |
| *** alexchadin has quit IRC | 09:17 | |
| *** tssurya has joined #openstack-meeting-alt | 09:17 | |
| *** fzdarsky has joined #openstack-meeting-alt | 09:21 | |
| *** bfernando has joined #openstack-meeting-alt | 09:24 | |
| *** dsariel has joined #openstack-meeting-alt | 09:25 | |
| *** ssathaye has quit IRC | 09:26 | |
| *** ssathaye has joined #openstack-meeting-alt | 09:27 | |
| *** ykarel|lunch is now known as ykarel | 09:28 | |
| *** alexchadin has joined #openstack-meeting-alt | 09:36 | |
| *** derekh has joined #openstack-meeting-alt | 09:36 | |
| *** absubram has quit IRC | 09:44 | |
| *** zhurong has quit IRC | 09:46 | |
| *** kumarmn has joined #openstack-meeting-alt | 09:46 | |
| *** ganso has joined #openstack-meeting-alt | 09:49 | |
| *** kumarmn has quit IRC | 09:51 | |
| *** fzdarsky has quit IRC | 09:51 | |
| *** lpetrut has joined #openstack-meeting-alt | 09:52 | |
| *** armaan has quit IRC | 09:55 | |
| *** armaan has joined #openstack-meeting-alt | 09:56 | |
| *** hieulq has joined #openstack-meeting-alt | 09:59 | |
| *** daidv has quit IRC | 10:06 | |
| *** matrohon has quit IRC | 10:12 | |
| *** matrohon has joined #openstack-meeting-alt | 10:12 | |
| *** caowei has quit IRC | 10:21 | |
| *** pbourke has quit IRC | 10:21 | |
| *** pbourke has joined #openstack-meeting-alt | 10:23 | |
| *** markvoelker has joined #openstack-meeting-alt | 10:24 | |
| *** sambetts|afk is now known as sambetts | 10:26 | |
| *** panda is now known as panda|ruck | 10:26 | |
| *** armaan has quit IRC | 10:33 | |
| *** armaan has joined #openstack-meeting-alt | 10:34 | |
| *** alexchadin has quit IRC | 10:36 | |
| *** alexchadin has joined #openstack-meeting-alt | 10:36 | |
| *** pgadiya has joined #openstack-meeting-alt | 10:40 | |
| *** anilvenkata has quit IRC | 10:46 | |
| *** kumarmn has joined #openstack-meeting-alt | 10:47 | |
| *** kumarmn has quit IRC | 10:53 | |
| *** alexchadin has quit IRC | 10:54 | |
| *** alexchadin has joined #openstack-meeting-alt | 10:55 | |
| *** markvoelker has quit IRC | 10:57 | |
| *** anilvenkata has joined #openstack-meeting-alt | 10:58 | |
| *** alexchadin has quit IRC | 11:00 | |
| *** alexchadin has joined #openstack-meeting-alt | 11:09 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** yamamoto has joined #openstack-meeting-alt | 11:11 | |
| *** alexchadin has quit IRC | 11:22 | |
| *** alexchadin has joined #openstack-meeting-alt | 11:23 | |
| *** matrohon has quit IRC | 11:26 | |
| *** yamamoto_ has joined #openstack-meeting-alt | 11:26 | |
| *** yamamoto has quit IRC | 11:29 | |
| *** sdague has joined #openstack-meeting-alt | 11:41 | |
| *** matrohon has joined #openstack-meeting-alt | 11:42 | |
| *** kumarmn has joined #openstack-meeting-alt | 11:48 | |
| *** kumarmn has quit IRC | 11:53 | |
| *** julim has quit IRC | 11:55 | |
| *** alexchadin has quit IRC | 11:55 | |
| *** alexchadin has joined #openstack-meeting-alt | 11:57 | |
| *** fzdarsky has joined #openstack-meeting-alt | 12:07 | |
| *** jkilpatr has joined #openstack-meeting-alt | 12:15 | |
| *** tesseract has joined #openstack-meeting-alt | 12:15 | |
| *** jkilpatr has quit IRC | 12:19 | |
| *** jkilpatr has joined #openstack-meeting-alt | 12:19 | |
| *** raildo has joined #openstack-meeting-alt | 12:20 | |
| *** anilvenkata has quit IRC | 12:20 | |
| *** oidgar has joined #openstack-meeting-alt | 12:28 | |
| *** fzdarsky is now known as fzdarsky|afk | 12:30 | |
| *** fzdarsky|afk has quit IRC | 12:30 | |
| *** anilvenkata has joined #openstack-meeting-alt | 12:37 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** panda|ruck is now known as panda|ruck|lunch | 12:38 | |
| *** julim has joined #openstack-meeting-alt | 12:39 | |
| *** slagle has joined #openstack-meeting-alt | 12:47 | |
| *** slagle has joined #openstack-meeting-alt | 12:47 | |
| *** kumarmn has joined #openstack-meeting-alt | 12:49 | |
| *** andreaf has quit IRC | 12:53 | |
| *** andreaf has joined #openstack-meeting-alt | 12:53 | |
| *** kumarmn has quit IRC | 12:54 | |
| *** markvoelker has joined #openstack-meeting-alt | 12:54 | |
| *** raildo has quit IRC | 13:04 | |
| *** jaypipes has joined #openstack-meeting-alt | 13:08 | |
| *** dprince has joined #openstack-meeting-alt | 13:08 | |
| *** ethfci has joined #openstack-meeting-alt | 13:09 | |
| *** dtrainor has joined #openstack-meeting-alt | 13:09 | |
| *** panda|ruck|lunch is now known as panda|ruck | 13:13 | |
| *** fzdarsky has joined #openstack-meeting-alt | 13:15 | |
| *** danpawlik_ has quit IRC | 13:17 | |
| *** alexchadin has quit IRC | 13:19 | |
| *** raildo has joined #openstack-meeting-alt | 13:20 | |
| *** armaan has quit IRC | 13:21 | |
| *** armaan has joined #openstack-meeting-alt | 13:21 | |
| *** yangyapeng has quit IRC | 13:26 | |
| *** yangyapeng has joined #openstack-meeting-alt | 13:27 | |
| *** raildo has quit IRC | 13:28 | |
| *** markvoelker has quit IRC | 13:28 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** janki has quit IRC | 13:31 | |
| *** janki has joined #openstack-meeting-alt | 13:32 | |
| *** yangyapeng has quit IRC | 13:32 | |
| *** krtaylor has quit IRC | 13:33 | |
| *** lennyb_ has joined #openstack-meeting-alt | 13:33 | |
| *** lennyb_ has quit IRC | 13:34 | |
| *** jcoufal has joined #openstack-meeting-alt | 13:34 | |
| *** matrohon has quit IRC | 13:36 | |
| *** armaan has quit IRC | 13:36 | |
| *** matrohon has joined #openstack-meeting-alt | 13:36 | |
| *** armaan has joined #openstack-meeting-alt | 13:37 | |
| *** julim has quit IRC | 13:37 | |
| *** julim has joined #openstack-meeting-alt | 13:37 | |
| *** danpawlik has joined #openstack-meeting-alt | 13:37 | |
| *** julim has quit IRC | 13:38 | |
| *** raildo has joined #openstack-meeting-alt | 13:43 | |
| *** ttsiouts has joined #openstack-meeting-alt | 13:44 | |
| *** ykarel is now known as ykarel|away | 13:46 | |
| *** kumarmn has joined #openstack-meeting-alt | 13:50 | |
| *** finucannot is now known as stephenfin | 13:50 | |
| *** cdent has joined #openstack-meeting-alt | 13:52 | |
| *** caowei has joined #openstack-meeting-alt | 13:53 | |
| *** kumarmn has quit IRC | 13:54 | |
| *** takashin has joined #openstack-meeting-alt | 13:56 | |
| *** ykarel|away has quit IRC | 13:57 | |
| *** krtaylor has joined #openstack-meeting-alt | 13:57 | |
| *** yangyapeng has joined #openstack-meeting-alt | 13:59 | |
| *** efried has joined #openstack-meeting-alt | 13:59 | |
| efried | #startmeeting nova_scheduler | 14:00 |
|---|---|---|
| openstack | Meeting started Mon Jan 15 14:00:01 2018 UTC and is due to finish in 60 minutes. The chair is efried. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: nova_scheduler)" | 14:00 | |
| openstack | The meeting name has been set to 'nova_scheduler' | 14:00 |
| jaypipes | o/ | 14:00 |
| takashin | o/ | 14:00 |
| ttsiouts | o/ | 14:00 |
| cdent | o/ | 14:00 |
| efried | Let's do this. | 14:00 |
| efried | #topic Reviews: Nested Resource Providers | 14:01 |
| *** openstack changes topic to "Reviews: Nested Resource Providers (Meeting topic: nova_scheduler)" | 14:01 | |
| efried | ComputeDriver.update_provider_tree() series starting with: https://review.openstack.org/#/c/532563/ (efried) | 14:01 |
| jaypipes | efried: I will get back to those reviews today | 14:01 |
| efried | jaypipes Thanks. Actually the most urgent thing from you is what's queued up for open discussion. | 14:02 |
| efried | For the above, I need to tweak the current bottom of the series per cdent comments. | 14:02 |
| efried | It should be noted that I recently reordered that guy to get set_traits_for_provider done earlier, because mgoddard needs that for the ironic-traits bp. | 14:03 |
| efried | Nested affordance in GET /allocation_candidates series starting with: https://review.openstack.org/#/c/531443/1 (jaypipes) | 14:03 |
| efried | Poor jaypipes has been laid up for the past week. | 14:03 |
| *** jkilpatr has quit IRC | 14:03 | |
| efried | The above needs to be rebased with some refactoring in light of test helpers which (finally!) merged. | 14:04 |
| jaypipes | yeah, sorry about that :( | 14:04 |
| jaypipes | will do today. | 14:04 |
| jaypipes | I noticed from the small amount of email checking I did last week that the gate looked fairly horrible... | 14:04 |
| efried | Granular resource requests (efried): https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/granular-resource-requests (waiting for ^^) | 14:04 |
| efried | Anything on NRP? | 14:05 |
| efried | #topic Reviews: Alternate Hosts | 14:05 |
| *** openstack changes topic to "Reviews: Alternate Hosts (Meeting topic: nova_scheduler)" | 14:05 | |
| efried | We're down to Just One More Patch mode here. https://review.openstack.org/526436 looks like it needs a pep8 fix. | 14:06 |
| *** jkilpatr has joined #openstack-meeting-alt | 14:06 | |
| efried | Anything on alt hosts? | 14:06 |
| efried | #topic Placement bugs | 14:07 |
| *** openstack changes topic to "Placement bugs (Meeting topic: nova_scheduler)" | 14:07 | |
| efried | https://bugs.launchpad.net/nova/+bugs?field.tag=placement | 14:07 |
| efried | We should plan on doing a leetle scrubbyscrub shortly after ff | 14:07 |
| cdent | I made a new bug about unnecessary imports | 14:07 |
| cdent | #link imports: https://bugs.launchpad.net/nova/+bug/1743120 | 14:07 |
| openstack | Launchpad bug 1743120 in OpenStack Compute (nova) "placement inadvertently imports many python modules it does not need" [Low,Triaged] - Assigned to Chris Dent (cdent) | 14:07 |
| cdent | and I'm working on some changes to fix it | 14:08 |
| efried | neat | 14:08 |
| efried | Anything else on bugs, placement or otherwise-scheduler? | 14:08 |
| jaypipes | nothing from me, post flu-ness. | 14:09 |
| efried | #topic Open discussion | 14:09 |
| *** openstack changes topic to "Open discussion (Meeting topic: nova_scheduler)" | 14:09 | |
| efried | One item on the agenda | 14:09 |
| efried | ProviderTree accessors (see https://review.openstack.org/#/c/533244/) | 14:09 |
| efried | Mainly needing jaypipes to weigh in here before I get too much further. | 14:09 |
| efried | Assuming you're not yet caught up on emails and whatnot, here's the executive summary: | 14:10 |
| efried | In order to have a way to compare the ProviderTree I get back from ComputeDriver.update_provider_tree with the one cached in report client, I need to be able to get to the actual fields in the _Provider | 14:10 |
| efried | The above patch is one possible way of doing it, that still protects the privacy/integrity of _Provider. | 14:11 |
| efried | Another way would be individual getters at the ProviderTree level for each attribute of _Provider. So like get_traits_for(name_or_uuid) etc. | 14:11 |
| efried | My plan is to start implementing that method that diffs and flushes ProviderTree changes today. | 14:12 |
| *** pgadiya has quit IRC | 14:12 | |
| efried | So whereas I don't need +2s or anything, I'd at least like general buy-in on the approach. | 14:12 |
| * efried hands the mic to jaypipes | 14:13 | |
| jaypipes | efried: I was thinking more along the lines of a ProviderTree.delta(ProviderTree) method which would return a list of dicts of changed provider information | 14:13 |
| jaypipes | efried: and that ProviderTree.delta() method would only be used/called by the reportclient once the update_provider_tree() virt driver method returns True (indicating *something* was modified in the tree) | 14:14 |
| efried | jaypipes Even that method would be ProviderTree needing to get at the individual fields of _Provider. | 14:14 |
| jaypipes | efried: of the other ProviderTree's _Provider(s), yes. | 14:15 |
| efried | I.e. what we're talking about is at a lower level than ProviderTree.delta (or whatever) | 14:15 |
| jaypipes | efried: understood. | 14:16 |
| efried | So the proposed patch is one way of doing that: it basically provides a copyout of a podo version of _Provider | 14:16 |
| jaypipes | efried: so ProviderTree.delta() would need to call this ProviderTree.snapshot() method and operate against those snapshots of the other tree's _Provider objects | 14:16 |
| efried | jaypipes Just so. | 14:17 |
| jaypipes | efried: yeah, I get it. | 14:17 |
| cdent | (there ya go efried, that's some of the additional context for your commit message...) | 14:17 |
| efried | So the question is: you cool with the snapshot idea or do we go for a more bean-like set of getters? | 14:18 |
| efried | cdent ack | 14:18 |
| jaypipes | efried: you could also do a ProviderTree.clone() method that would return a new identical ProviderTree that the delta() method would then fully own (and not have to worry about potentially other threads writing to.) | 14:18 |
| jaypipes | efried: definitely like the snapshot approach over a bean-like getter approach. | 14:19 |
| *** rossella_s has quit IRC | 14:19 | |
| jaypipes | efried: I also think a clone() approach would be more Pythonic, though -- in the mould of copy.deepcopy()-like semantics against dicts | 14:19 |
| efried | jaypipes Yes; I wasn't as much a fan of simple clone (deepcopy) because it still breaks the (albeit nominal) privacy boundary. | 14:19 |
| jaypipes | what does edleafe think? or should I read the reviews? :) | 14:19 |
| efried | jaypipes Well, edleafe was involved in the brainstorm on Friday. Not much beyond that in the review. | 14:20 |
| jaypipes | efried: well, I see his reviews on PS1 | 14:20 |
| jaypipes | efried: in short, I think the snapshot approach is perfectly fine. | 14:21 |
| jaypipes | efried: it does one job and accomplishes that job. | 14:21 |
| efried | jaypipes Great, I will proceed. | 14:21 |
| jaypipes | efried: let's move forward, I think | 14:21 |
| efried | That's everything on the agenda. Anyone have anything else to discuss? | 14:21 |
| *** rossella_s has joined #openstack-meeting-alt | 14:21 | |
| jaypipes | cdent: what were your thoughts on the snapshot/clone thing? | 14:22 |
| cdent | I'm okay with now that I've read the additional context here. It wasn't clear what the calling context was going to be, but now that it is more clear, I think a read only snapshot is a reasonable way to go, if that level of protect is important (I'm still not _entirely_ clear why it is) | 14:23 |
| efried | Realistically, the ProviderTree given to and returned by ComputeDriver.update_provider_tree doesn't need all this careful locking at all, ever. | 14:24 |
| efried | But IMO having the code take that for granted and violate the privacy markings of _Provider is not a good approach. | 14:24 |
| jaypipes | efried: never say never.. | 14:25 |
| efried | Exactly. And those protections *are* necessary for the ProviderTree that report client uses as a cache. | 14:25 |
| jaypipes | ya | 14:25 |
| efried | So we need *something* there. And this is about as lightweight/unintrusive as I can think of. | 14:26 |
| cdent | if we need something, then yeah, seems okay to me, it is pretty clean and comprehensible | 14:26 |
| efried | Good deal. Thanks y'all. | 14:26 |
| efried | Anything else? | 14:27 |
| jaypipes | 42 | 14:27 |
| cdent | WRONG | 14:27 |
| jaypipes | heh | 14:27 |
| efried | Okay, thanks everyone. | 14:27 |
| efried | #endmeeting | 14:27 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:27 | |
| openstack | Meeting ended Mon Jan 15 14:27:40 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:27 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/nova_scheduler/2018/nova_scheduler.2018-01-15-14.00.html | 14:27 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/nova_scheduler/2018/nova_scheduler.2018-01-15-14.00.txt | 14:27 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/nova_scheduler/2018/nova_scheduler.2018-01-15-14.00.log.html | 14:27 |
| cdent | thanks efried | 14:27 |
| * cdent goes home | 14:27 | |
| *** cdent has quit IRC | 14:28 | |
| *** kumarmn has joined #openstack-meeting-alt | 14:30 | |
| *** kumarmn_ has joined #openstack-meeting-alt | 14:30 | |
| *** takashin has left #openstack-meeting-alt | 14:33 | |
| *** kumarmn has quit IRC | 14:34 | |
| *** tssurya has quit IRC | 14:35 | |
| *** matrohon has quit IRC | 14:41 | |
| *** matrohon has joined #openstack-meeting-alt | 14:41 | |
| *** gcb has joined #openstack-meeting-alt | 14:44 | |
| *** caowei has quit IRC | 14:45 | |
| *** ameeda has joined #openstack-meeting-alt | 14:48 | |
| *** ameeda has left #openstack-meeting-alt | 14:48 | |
| *** tssurya has joined #openstack-meeting-alt | 14:50 | |
| *** jkilpatr has quit IRC | 14:54 | |
| *** jkilpatr has joined #openstack-meeting-alt | 14:55 | |
| *** matrohon has quit IRC | 14:58 | |
| *** yamahata has joined #openstack-meeting-alt | 14:59 | |
| *** Leo_m has joined #openstack-meeting-alt | 15:03 | |
| *** yamamoto_ has quit IRC | 15:06 | |
| *** oidgar has quit IRC | 15:12 | |
| *** armaan has quit IRC | 15:13 | |
| *** chhavi__ has joined #openstack-meeting-alt | 15:13 | |
| *** chhagarw has joined #openstack-meeting-alt | 15:13 | |
| *** tssurya has quit IRC | 15:15 | |
| *** julim has joined #openstack-meeting-alt | 15:20 | |
| *** links has quit IRC | 15:20 | |
| *** markvoelker has joined #openstack-meeting-alt | 15:25 | |
| *** oidgar has joined #openstack-meeting-alt | 15:26 | |
| *** tssurya has joined #openstack-meeting-alt | 15:28 | |
| *** matrohon has joined #openstack-meeting-alt | 15:28 | |
| *** hongbin has joined #openstack-meeting-alt | 15:29 | |
| *** dsariel has quit IRC | 15:38 | |
| *** dsariel has joined #openstack-meeting-alt | 15:48 | |
| *** felipemonteiro has joined #openstack-meeting-alt | 15:50 | |
| *** sxc731_ has joined #openstack-meeting-alt | 15:52 | |
| *** gcb has quit IRC | 15:54 | |
| *** felipemonteiro_ has joined #openstack-meeting-alt | 15:54 | |
| *** sxc731_ has quit IRC | 15:56 | |
| *** felipemonteiro has quit IRC | 15:58 | |
| *** markvoelker has quit IRC | 15:59 | |
| *** sxc731_ has joined #openstack-meeting-alt | 15:59 | |
| *** chyka has joined #openstack-meeting-alt | 16:01 | |
| *** chyka has quit IRC | 16:02 | |
| *** chyka has joined #openstack-meeting-alt | 16:02 | |
| *** yamamoto has joined #openstack-meeting-alt | 16:07 | |
| *** yamamoto has quit IRC | 16:14 | |
| *** anilvenkata has quit IRC | 16:16 | |
| *** sxc731_ has quit IRC | 16:16 | |
| *** sxc731_ has joined #openstack-meeting-alt | 16:16 | |
| *** sxc731_ has quit IRC | 16:20 | |
| *** anilvenkata has joined #openstack-meeting-alt | 16:22 | |
| *** sxc731_ has joined #openstack-meeting-alt | 16:23 | |
| *** sridharg has quit IRC | 16:27 | |
| *** dave-mcc_ has joined #openstack-meeting-alt | 16:29 | |
| *** rossella_s has quit IRC | 16:38 | |
| *** erlon has joined #openstack-meeting-alt | 16:39 | |
| *** rossella_s has joined #openstack-meeting-alt | 16:40 | |
| *** tesseract has quit IRC | 16:40 | |
| *** gouthamr has joined #openstack-meeting-alt | 16:42 | |
| *** gyee has joined #openstack-meeting-alt | 16:43 | |
| *** jpena is now known as jpena|brb | 16:44 | |
| *** markvoelker has joined #openstack-meeting-alt | 16:47 | |
| *** gouthamr has quit IRC | 16:50 | |
| *** gouthamr has joined #openstack-meeting-alt | 16:57 | |
| *** marios has quit IRC | 17:02 | |
| *** sxc731_ has quit IRC | 17:02 | |
| *** oikiki has joined #openstack-meeting-alt | 17:03 | |
| *** chhavi__ has quit IRC | 17:03 | |
| *** chhagarw has quit IRC | 17:03 | |
| *** armaan has joined #openstack-meeting-alt | 17:06 | |
| *** dave-mcc_ has quit IRC | 17:08 | |
| *** felipemonteiro_ has quit IRC | 17:10 | |
| *** rossella_s has quit IRC | 17:15 | |
| *** rossella_s has joined #openstack-meeting-alt | 17:16 | |
| *** janki has quit IRC | 17:17 | |
| *** efried is now known as efried_rollin | 17:19 | |
| *** links has joined #openstack-meeting-alt | 17:21 | |
| *** matrohon has quit IRC | 17:24 | |
| *** jpena|brb is now known as jpena | 17:27 | |
| *** links has quit IRC | 17:50 | |
| *** rossella_s has quit IRC | 17:55 | |
| *** rossella_s has joined #openstack-meeting-alt | 17:58 | |
| *** bfernando has quit IRC | 18:02 | |
| *** derekh has quit IRC | 18:02 | |
| *** armaan has quit IRC | 18:04 | |
| *** jkilpatr has quit IRC | 18:05 | |
| *** sambetts is now known as sambetts|afk | 18:14 | |
| *** jpena is now known as jpena|off | 18:18 | |
| *** jkilpatr has joined #openstack-meeting-alt | 18:18 | |
| *** kmalloc has joined #openstack-meeting-alt | 18:19 | |
| *** Swami_ has joined #openstack-meeting-alt | 18:20 | |
| *** gouthamr has quit IRC | 18:20 | |
| *** corvus is now known as jeblair | 18:21 | |
| *** jeblair is now known as corvus | 18:21 | |
| *** lpetrut has quit IRC | 18:22 | |
| *** rossella_s has quit IRC | 18:25 | |
| *** Leo_m has quit IRC | 18:26 | |
| *** rossella_s has joined #openstack-meeting-alt | 18:26 | |
| *** rossella_s has quit IRC | 18:31 | |
| *** rossella_s has joined #openstack-meeting-alt | 18:32 | |
| *** felipemonteiro has joined #openstack-meeting-alt | 18:34 | |
| *** felipemonteiro_ has joined #openstack-meeting-alt | 18:35 | |
| *** rossella_s has quit IRC | 18:37 | |
| *** myoung|pto is now known as myoung | 18:39 | |
| *** felipemonteiro has quit IRC | 18:39 | |
| *** markstur has joined #openstack-meeting-alt | 18:39 | |
| *** rossella_s has joined #openstack-meeting-alt | 18:41 | |
| *** oidgar has quit IRC | 18:46 | |
| *** dsariel has quit IRC | 18:51 | |
| *** Leo_m has joined #openstack-meeting-alt | 18:57 | |
| *** corvus is now known as jeblair | 18:58 | |
| *** jeblair is now known as corvus | 18:58 | |
| *** Leo_m has quit IRC | 19:01 | |
| *** Leo_m has joined #openstack-meeting-alt | 19:12 | |
| *** Leo_m has quit IRC | 19:16 | |
| *** felipemonteiro__ has joined #openstack-meeting-alt | 19:17 | |
| *** edleafe- has joined #openstack-meeting-alt | 19:17 | |
| *** edleafe has quit IRC | 19:19 | |
| *** edleafe- is now known as edleafe | 19:19 | |
| *** felipemonteiro_ has quit IRC | 19:20 | |
| *** fnaval has joined #openstack-meeting-alt | 19:27 | |
| *** Leo_m has joined #openstack-meeting-alt | 19:27 | |
| *** fnaval_ has joined #openstack-meeting-alt | 19:28 | |
| *** fnaval_ has quit IRC | 19:29 | |
| *** Leo_m_ has joined #openstack-meeting-alt | 19:31 | |
| *** oikiki has quit IRC | 19:32 | |
| *** oikiki has joined #openstack-meeting-alt | 19:32 | |
| *** Leo_m has quit IRC | 19:32 | |
| *** fnaval has quit IRC | 19:32 | |
| *** fnaval has joined #openstack-meeting-alt | 19:33 | |
| *** fnaval has quit IRC | 19:33 | |
| *** chyka has quit IRC | 19:42 | |
| *** rossella_s has quit IRC | 19:48 | |
| *** rossella_s has joined #openstack-meeting-alt | 19:50 | |
| *** jkilpatr has quit IRC | 19:55 | |
| *** efried_rollin is now known as efried | 19:55 | |
| *** tobiash has joined #openstack-meeting-alt | 20:02 | |
| *** slaweq has joined #openstack-meeting-alt | 20:15 | |
| *** rmascena has joined #openstack-meeting-alt | 20:19 | |
| *** lpetrut has joined #openstack-meeting-alt | 20:19 | |
| *** raildo has quit IRC | 20:21 | |
| *** dansmith has quit IRC | 20:22 | |
| *** markvoelker has quit IRC | 20:22 | |
| *** fnaval has joined #openstack-meeting-alt | 20:27 | |
| *** matrohon has joined #openstack-meeting-alt | 20:34 | |
| *** dansmith has joined #openstack-meeting-alt | 20:35 | |
| *** dansmith is now known as Guest49522 | 20:35 | |
| *** Guest49522 is now known as dansmith | 20:40 | |
| *** matrohon has quit IRC | 20:45 | |
| *** matrohon has joined #openstack-meeting-alt | 20:46 | |
| *** fnaval has quit IRC | 20:52 | |
| *** jkilpatr has joined #openstack-meeting-alt | 20:53 | |
| *** lpetrut has quit IRC | 20:56 | |
| *** chyka has joined #openstack-meeting-alt | 20:59 | |
| *** fnaval has joined #openstack-meeting-alt | 21:03 | |
| *** dprince has quit IRC | 21:06 | |
| *** Leo_m_ has quit IRC | 21:14 | |
| *** Leo_m has joined #openstack-meeting-alt | 21:19 | |
| *** ganso has quit IRC | 21:29 | |
| *** dave-mccowan has joined #openstack-meeting-alt | 21:31 | |
| *** dgonzalez has quit IRC | 21:33 | |
| *** erlon has quit IRC | 21:39 | |
| *** dgonzalez has joined #openstack-meeting-alt | 21:42 | |
| *** lbragstad has quit IRC | 21:44 | |
| *** oikiki has quit IRC | 21:44 | |
| *** oikiki has joined #openstack-meeting-alt | 21:45 | |
| *** lbragstad has joined #openstack-meeting-alt | 21:47 | |
| *** matrohon has quit IRC | 21:48 | |
| *** erlon has joined #openstack-meeting-alt | 21:48 | |
| *** fzdarsky is now known as fzdarsky|afk | 21:50 | |
| *** chyka_ has joined #openstack-meeting-alt | 21:50 | |
| *** chyka has quit IRC | 21:53 | |
| *** armaan has joined #openstack-meeting-alt | 21:55 | |
| clarkb | zuul meeting time? | 22:01 |
| dmsimard | ohaui | 22:01 |
| corvus | anyone else around? | 22:02 |
| *** matrohon has joined #openstack-meeting-alt | 22:03 | |
| fungi | moi | 22:04 |
| clarkb | I'm going to have to cut my meeting short to run some errands. But have some last minute nodepool stuff (zuul too IGuess) | 22:04 |
| *** tpsilva has quit IRC | 22:04 | |
| corvus | #startmeeting zuul | 22:04 |
| openstack | Meeting started Mon Jan 15 22:04:49 2018 UTC and is due to finish in 60 minutes. The chair is corvus. Information about MeetBot at http://wiki.debian.org/MeetBot. | 22:04 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 22:04 |
| *** openstack changes topic to " (Meeting topic: zuul)" | 22:04 | |
| openstack | The meeting name has been set to 'zuul' | 22:04 |
| corvus | #topic Roadmap | 22:05 |
| *** openstack changes topic to "Roadmap (Meeting topic: zuul)" | 22:05 | |
| corvus | I'm going to ping folks individually this week to check up on status | 22:05 |
| corvus | but does anyone here working on a 3.0 release blocker have an issue we should talk about now? | 22:05 |
| corvus | (i know a lot of folks are afk today, but i thought i'd ask) | 22:05 |
| corvus | #topic RAM governor for the executors | 22:06 |
| *** openstack changes topic to "RAM governor for the executors (Meeting topic: zuul)" | 22:06 | |
| corvus | dmsimard: i think this is your topic? | 22:07 |
| dmsimard | oh, from last week yes | 22:07 |
| corvus | #link https://review.openstack.org/508960 ram governor | 22:08 |
| dmsimard | We're generally memory constrained right now -- we're often finding zuul executors in swap territory and at that point it becomes a vicious circle quickly (can't clear jobs fast enough, so you get more jobs, etc.) and there's several OOM killers going around | 22:08 |
| dmsimard | So we want to land and enable the RAM governor ASAP but there's also another "governor" I'd like to talk about -- it'd be "max concurrent builds" | 22:09 |
| corvus | dmsimard: when executors go above a certain load average they shouldn't accept new jobs | 22:09 |
| clarkb | on the scheduler side the memory consumer is the size of the zuul config model. Do we know what is consuming the memory on executors? is it ansible? | 22:09 |
| dmsimard | Regardless of our current governors (even pretending RAM had landed), there's nothing preventing a single executor from picking up 200 builds by itself | 22:09 |
| corvus | dmsimard: when have you seeen executors accept new jobs because they can't clear them fast enough? | 22:09 |
| corvus | dmsimard: yes there is -- we would have two things preventing it -- a load governor and a ram governor | 22:10 |
| corvus | right now we have one | 22:10 |
| clarkb | (just want to amke sure that governing job exectution is expected to reduce memory use and it isn't the finger daemon that is consuming all the memroy for example) | 22:10 |
| dmsimard | corvus: not from a cold boot -- when all executors crashed a week ago, ze01 started first and picked up all the backlogged jobs and (eventually) loaded up to 150 | 22:10 |
| *** lbragstad has quit IRC | 22:10 | |
| fungi | i gather the issue with the system load governor not kicking in fast enough is that system load average is a trailing indicator and so can in certain thundering herd scenarios pick up a glut of jobs before the system load spikes high enough to stop it | 22:11 |
| corvus | clarkb: i *think* it's ansible eating the memory, but it's not leaking, it just uses a lot. at least, that's my recollection. it would be good to confirm that. | 22:11 |
| corvus | dmsimard: yes, that's true. i think after we have a ram governor, we should look into tuning the rate at which jobs are accepted. | 22:11 |
| fungi | that sounds sane | 22:12 |
| dmsimard | Generally speaking, there is only so many SSH connections/ansible playbooks we can have running at any given time | 22:12 |
| dmsimard | Wouldn't it be reasonable to say an executor can accept no more than 100 concurrent builds for example ? | 22:12 |
| corvus | dmsimard: i'd like to save 'max jobs per server' as a last resort -- as in, i'd like us to avoid ever implementing it if possible, unless we completely fail at everything else. the reason is that not all jobs are equal in resource usage. i think it would be best if the executors could regulate themselves toward maximum resource usage without going overboard. | 22:13 |
| fungi | dmsimard: depending on the resources available and performance of the server, that number may vary quite a lot though right? | 22:13 |
| fungi | what if you have two executors, one of which is ~half as powerful as the other... having zuul scale itself to them based on available resources is nice | 22:13 |
| dmsimard | fungi: If something like that lands, it would be something configurable (with a sane default) imo | 22:14 |
| dmsimard | the way I see it, it's more of a safety net | 22:14 |
| *** florianf has quit IRC | 22:14 | |
| fungi | not enough of a safety net unless you get into fiddling with per-server knobs rather than having a sane resource scheduler which can guess the right values for you | 22:15 |
| corvus | i don't want admins to have to tune these options. there is no sensible global default for max jobs per server, and would need to be always individually tuned. further, that ignores that not all jobs are the same, so it's problematic. | 22:15 |
| corvus | a job with 10 nodes that runs for 3 hours is different than a job with zero nodes that runs for 30 seconds. both are very likely in the same system. | 22:15 |
| dmsimard | right | 22:16 |
| pabelanger | o/ sorry I am late | 22:16 |
| dmsimard | pabelanger: ohai I was actually about to ask, do we think we can land https://review.openstack.org/#/c/508960/ soon ? | 22:16 |
| corvus | i agree that we need to prevent the hysterisis from happening -- i think the road there goes through the ram governor first, then tune the acceptance rate (there should already be a small rolloff, but maybe we need to adjust that a bit) so that the trailing indicators have more time to catch up. finally, we may want to tune our heuristics a bit to give the trailing indicators more headroom. | 22:17 |
| pabelanger | dmsimard: i think we want to add tests first, I'm hoping to finish that up in the next day or so | 22:17 |
| dmsimard | corvus: fwiw I agree that the max build idea is not a definitive answer and instead we might want to do like you mentioned and revisit/tune how executors pick up jobs in the first place | 22:18 |
| corvus | pabelanger: ++ we should be able to use mock to return some ram data | 22:18 |
| pabelanger | corvus: wfm | 22:18 |
| *** lbragstad has joined #openstack-meeting-alt | 22:19 | |
| corvus | dmsimard: it looks like right now we delay job acceptance a small amount but only with the goal of spreading the load across executors, so the response time is still pretty quick | 22:19 |
| pabelanger | speaking of jobs, one thing zuulv2.5 did, and I don't believe zuulv3 does, is we had some sort of back off method so a single executor wouldn't accept a bunch of jobs at once. That seem to work well in zuulv2.5 with our zuul-launchers | 22:20 |
| corvus | and it only looks at the number of jobs currently running | 22:20 |
| corvus | what we may want to do is adjust that to *also* factor in how recently a job was accepted | 22:20 |
| *** matrohon has quit IRC | 22:21 | |
| corvus | or just increase the delay that's already there and only use jobs running | 22:21 |
| *** rcernin has joined #openstack-meeting-alt | 22:21 | |
| corvus | it's currently: delay = (workers ** 2) / 1000.0 | 22:21 |
| corvus | 'workers' means jobs running in this context | 22:21 |
| fungi | would that also explain why when all the executors crashed at once, the first one to get started went nuts on the backlog? | 22:21 |
| fungi | since there wasn't even the rotation between executors to save it | 22:22 |
| clarkb | ya jobs running seems like maybe a better option than workers | 22:22 |
| corvus | that's only going to slow us down 6.4 seconds with 80 jobs, so that's not enough time for load/ram to start to catch up | 22:22 |
| corvus | clarkb: no sorry, the variable is called "workers" but it means "number of jobs that this executor is running" | 22:23 |
| corvus | it's the internal executor gearman worker count | 22:23 |
| fungi | fair. and the load governor is based on one-minute load average, so you have a lot of time to ramp up to untenable levels of activity | 22:23 |
| clarkb | gotcha | 22:23 |
| *** slaweq has quit IRC | 22:24 | |
| fungi | i have similar worries about the ram governor, if the amount of ram ansible is going to use grows over time (we may take on a glut of jobs, and not finish old ones fast enough to make way for the memory growth of the new spike) | 22:24 |
| pabelanger | We also don't start running ansible right away now too, we first merge code into local executor. Perhaps that isn't load / memory heavy? | 22:24 |
| *** slaweq has joined #openstack-meeting-alt | 22:24 | |
| corvus | pabelanger: it's not too heavy, but it is a delay worth considering. we could even let that be a natural source of delay -- like don't acquire a new job until we've completed merging the most recent job. | 22:25 |
| dmsimard | pabelanger: I'm wondering if swap should be taken into account in the RAM governor (and how) | 22:25 |
| corvus | that would be probably be fine fully loaded, but it would make for a very slow start. | 22:25 |
| pabelanger | corvus: yah, that might be something to try. I like that | 22:26 |
| fungi | i think once we've started paging zuul activity out to swap space, it's already doomed | 22:26 |
| dmsimard | fungi: that's for scheduler but executors will keep running even when swapping | 22:26 |
| dmsimard | ideally the ram governor prevents us from reaching swap territory | 22:26 |
| fungi | how well do they keep running? | 22:26 |
| dmsimard | not well -- when the executors start swapping, execution becomes largely I/O bound and there's a higher percentage of I/O wait | 22:27 |
| fungi | if "keep running" means jobs start timing out because it takes too long for ansible to start the next task/playbook then that's basically it | 22:27 |
| corvus | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64003&rra_id=all | 22:28 |
| fungi | i didn't mean doomed to need a restart, i meant doomed to introduce otherwise avoidable job failures | 22:28 |
| dmsimard | fungi: I noticed the i/o wait and swap usage when I was trying to understand the SSH connection issues, there might be a correlation but I don't know. | 22:28 |
| *** slaweq has quit IRC | 22:29 | |
| corvus | interesting -- they're pretty active swapping but keep the used memory close to 50% | 22:29 |
| corvus | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64004&rra_id=all | 22:29 |
| corvus | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64005&rra_id=all | 22:29 |
| fungi | i wonder if buffer space there is the ansible stdout buffering stuff | 22:30 |
| *** armaan has quit IRC | 22:30 | |
| dmsimard | fungi: actually I asked #ansible-devel about that and the buffer is on the target node, not the control node | 22:30 |
| pabelanger | speaking of SSH connection issues, we could using SSH retries from ansible: https://review.openstack.org/512130/ to help add some fail protection to jobs | 22:31 |
| dmsimard | so it wouldn't explain the ram usage | 22:31 |
| corvus | dmsimard: did they indicate what happens when the target node sends the data back to the control node? | 22:31 |
| fungi | regardless, system and iowait cpu usage there don't look super healthy, leading me to wonder whether we still have too few executors at peak | 22:31 |
| pabelanger | maybe even expose it to be configurable some how to jobs | 22:31 |
| fungi | and 5-minute load average spiking up over 40 on ze01 just a few hours ago | 22:31 |
| dmsimard | pabelanger: there's some improvements we can do around SSH, yes | 22:31 |
| fungi | where it topped out around 4gb of swap in use | 22:32 |
| dmsimard | fungi: that's likely load due to i/o wait | 22:32 |
| dmsimard | (heavy swapping) | 22:32 |
| fungi | dmsimard: exactly | 22:32 |
| pabelanger | I also think, OSIC suggested some things we could also tune in ansible for network related issues. Need to see if I can find that etherpad | 22:32 |
| pabelanger | or was is OSA team | 22:32 |
| dmsimard | fungi: vm.swappinness is at 0 on ze01 too.. | 22:33 |
| dmsimard | I read that as "never swap ever" so I don't know what's going on | 22:33 |
| *** felipemonteiro__ has quit IRC | 22:33 | |
| dmsimard | Oh, actually it doesn't quite mean that | 22:33 |
| dmsimard | "Swap is disabled. In earlier versions, this meant that the kernel would swap only to avoid an out of memory condition, when free memory will be below vm.min_free_kbytes limit, but in later versions this is achieved by setting to 1." | 22:34 |
| fungi | dmsimard: no, it just means don't preemptively swap process memory out to make room for additional cache memory | 22:34 |
| dmsimard | and our min_free_kbytes is vm.min_free_kbytes = 11361 | 22:34 |
| fungi | these are fairly typical configuration for "cloud" virtual machines | 22:35 |
| corvus | fungi: our cpu usage from last week is significantly different from november: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64000&rra_id=4&view_type=&graph_start=1483002375&graph_end=1516055559 | 22:35 |
| clarkb | ok I've got to head out now. The two things I wanted to bring up were merging feature/zuulv3 to master. I tested this nodepool and wrote an infra list email about it. The other thing is my two nodepool changes to address cloud failures. 533771 and its parent. Left test split out as I expect it may need cleanup but should be good enough to show parent works | 22:35 |
| corvus | we may be seeing the hit from meltdown and may indeed need to add more executors | 22:35 |
| fungi | yep, it is a bit worse | 22:36 |
| corvus | clarkb: thanks, yeah, i think we can merge the branches soon, maybe let's set a date for thursday and send out a followup email? | 22:37 |
| fungi | meltdown mitigation performance hit seems as good a culprit as any | 22:37 |
| dmsimard | Re: adding more executors -- do we think we have the right size right now ? In terms of flavors, disk size, etc. | 22:38 |
| *** rmascena has quit IRC | 22:38 | |
| *** markstur has quit IRC | 22:38 | |
| pabelanger | looking at stats from zuul-launcher is a little interesting too: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=4683&rra_id=4&view_type=&graph_start=1483002628&graph_end=1516055812 | 22:38 |
| pabelanger | we do seem to be using more system resources with executors | 22:38 |
| corvus | pabelanger: that had a *very* different mode of operation | 22:38 |
| dmsimard | pabelanger: executors also run zuul-merger which is not negligible | 22:39 |
| pabelanger | yup | 22:39 |
| *** slaweq has joined #openstack-meeting-alt | 22:39 | |
| corvus | dmsimard: i'd argue it is negligible | 22:39 |
| corvus | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1519&rra_id=all | 22:40 |
| corvus | that's on a 2G server | 22:40 |
| dmsimard | but there's 8 zm nodes :P | 22:40 |
| corvus | yes, for paralellization | 22:41 |
| corvus | i'm just saying that the internal merger is not what's eating things up on the executors. we're just doing a lot more with ansible than we were in zuulv2.5 | 22:41 |
| * dmsimard nods | 22:41 | |
| corvus | (among other things, in zuul v2.5, we did *not* ship the entire console output back to the controlling ansible process) | 22:42 |
| corvus | anyway, to conclude: let's say the plan for now is: add ram governor, then slow job acceptance rate. sound good? | 22:43 |
| fungi | wfm | 22:44 |
| pabelanger | I think we also used 2.1.x vs 2.3.x, so possible ansible is now just using more resources | 22:44 |
| pabelanger | corvus: ++ | 22:44 |
| *** slaweq has quit IRC | 22:44 | |
| corvus | #agreed to reduce hysterisis and exceess resource usage: add ram governor, then slow job acceptance rate | 22:44 |
| corvus | #topic merging feature branches | 22:44 |
| *** openstack changes topic to "merging feature branches (Meeting topic: zuul)" | 22:44 | |
| corvus | we should be all prepared as clarkb said -- but we still may want to actually schedule this so no one is surprised | 22:45 |
| corvus | how about we say we'll do it on thursday? | 22:45 |
| fungi | the puppet changes to make that hitless for people using puppet-openstackci are merged at this point, right? | 22:46 |
| corvus | fungi: yep | 22:46 |
| pabelanger | thursday is fine by me | 22:46 |
| corvus | #link puppet-openstackci change https://review.openstack.org/523951 | 22:47 |
| fungi | i guess as long as the people using that module update it with some frequency they're protected. if they don't, then they're using a non-continuously-deployed puppet module to continuously deploy a service... so a learning experience for them? | 22:47 |
| pabelanger | we'll have to make some config changes to nodepool-builder, since it is using old syntax. I can propose some patches for that | 22:47 |
| corvus | fungi: yep. and i mean, it's not going to eat their data, they just need to checkout a different version and reinstall | 22:47 |
| pabelanger | maybe also upgrade to python3 at the same time | 22:47 |
| *** jcoufal has quit IRC | 22:48 | |
| corvus | pabelanger: heh, well, if we're checking out master on nodepool builders, then i think we'll automatically get switched to v3. :) | 22:49 |
| corvus | pabelanger: do you want to deploy new builders running from the feature/zuulv3 branch before we merge? | 22:50 |
| pabelanger | corvus: I was thinking maybe we first switch nodepool builders to feature/zuulv3 branch and get config file changes in place | 22:50 |
| pabelanger | yah | 22:50 |
| corvus | pabelanger: think that's reasonable to do before thursday? | 22:50 |
| corvus | hopefully there'll be more folks around tomorrow to help too | 22:50 |
| pabelanger | I believe so, I can start work on it ASAP | 22:50 |
| corvus | cool | 22:51 |
| corvus | #agreed schedule feature branch merge for thursday jan 18 | 22:51 |
| corvus | #topic open discussion | 22:51 |
| *** openstack changes topic to "open discussion (Meeting topic: zuul)" | 22:51 | |
| corvus | anything else? | 22:51 |
| fungi | oh, as far as release-related needs, i wouldn't mind if someone took a look at my change to link the zuul-base-jobs documentation from the user's guide | 22:52 |
| fungi | #link https://review.openstack.org/531912 Link to zuul-base-jobs docs from User's Guide | 22:52 |
| fungi | it's small, and could probably stand to be at least a little less small | 22:53 |
| *** markvoelker has joined #openstack-meeting-alt | 22:53 | |
| corvus | fungi: ++ | 22:53 |
| corvus | if that's it, i'll go ahead and end | 22:55 |
| *** chyka_ has quit IRC | 22:55 | |
| corvus | thanks everyone! | 22:55 |
| corvus | #endmeeting | 22:55 |
| fungi | thanks corvus! | 22:55 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:55 | |
| openstack | Meeting ended Mon Jan 15 22:55:45 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:55 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-01-15-22.04.html | 22:55 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-01-15-22.04.txt | 22:55 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-01-15-22.04.log.html | 22:55 |
| *** chyka has joined #openstack-meeting-alt | 22:56 | |
| *** markvoelker has quit IRC | 22:58 | |
| *** chhavi__ has joined #openstack-meeting-alt | 23:00 | |
| *** chhagarw has joined #openstack-meeting-alt | 23:00 | |
| *** lbragstad has quit IRC | 23:00 | |
| *** armaan has joined #openstack-meeting-alt | 23:01 | |
| *** hieulq has quit IRC | 23:02 | |
| *** hieulq has joined #openstack-meeting-alt | 23:03 | |
| *** chhavi__ has quit IRC | 23:04 | |
| *** chhagarw has quit IRC | 23:04 | |
| *** julim has quit IRC | 23:09 | |
| *** julim has joined #openstack-meeting-alt | 23:11 | |
| *** kumarmn_ has quit IRC | 23:17 | |
| *** kumarmn has joined #openstack-meeting-alt | 23:18 | |
| *** kumarmn has quit IRC | 23:22 | |
| *** Leo_m has quit IRC | 23:26 | |
| *** hongbin has quit IRC | 23:28 | |
| *** kumarmn has joined #openstack-meeting-alt | 23:35 | |
| *** kumarmn has quit IRC | 23:40 | |
| *** chyka_ has joined #openstack-meeting-alt | 23:44 | |
| *** chyka has quit IRC | 23:49 | |
| *** lbragstad has joined #openstack-meeting-alt | 23:53 | |
| *** erlon has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!