15:01:08 <Zara> #startmeeting storyboard
15:01:09 <openstack> Meeting started Wed Apr 20 15:01:08 2016 UTC and is due to finish in 60 minutes.  The chair is Zara. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:13 <openstack> The meeting name has been set to 'storyboard'
15:01:24 <Zara> #topic Announcements
15:01:36 <Zara> #info There will be no storyboard meeting next week!
15:01:44 <Zara> For we will all be at the summit!
15:02:07 <persia> At the task tracking session even
15:02:15 <Zara> yes!
15:02:18 <Zara> #link https://www.openstack.org/summit/austin-2016/summit-schedule/events/9171?goback=1
15:02:28 <Zara> it is the same day, I don't know how the time compares
15:02:33 <Zara> so keep an eye on the etherpad that day
15:02:47 <Zara> and the channel, probably a bit after the session
15:03:14 <Zara> please bear in mind that I am completely incapable of multitasking, so if I'm talking in person, I can't read irc
15:03:27 <Zara> hopefully other people on this team are a bit better at that one
15:03:44 <Zara> any other annoucements?
15:03:57 <SotK> _o_
15:04:40 <Zara> I think the session is 14:00 UTC
15:05:01 <Zara> there was a list with utc times that I've lost
15:05:07 <Zara> please ask in #storyboard for further info
15:05:26 <Zara> #topic Urgent Items
15:06:08 <Zara> we found the webclient hasn't been building for about a week, for storyboard.openstack.org
15:06:16 <Zara> it seems to be the same problem as last time
15:06:31 <Zara> ie: the workers failing to die when it's restarted
15:06:34 <Zara> until the machine gets full
15:06:46 <Zara> and puppet can't build because there isn't enough RAM free
15:06:56 <nibalizer> ya
15:07:03 <Zara> SotK has been investigating.
15:07:07 <nibalizer> does anyone know where the ram usage is coming from?
15:07:16 <nibalizer> is it time to just move storyboard.o.o to a bigger node?
15:07:21 * Zara passes the mic to SotK
15:07:28 <SotK> nibalizer: probably an inundation of storyboard-worker-daemon processes
15:07:34 <SotK> if not that, then I have no idea :)
15:08:21 <Zara> afaik, we believe there shouldn't be anywhere near that many processes, and it's probably a bug in some bit of the storyboard worker code
15:08:29 <SotK> yeah, there should be 5
15:08:47 <SotK> last time we ran out of ram its because there were 100-and-something
15:09:00 <Zara> so the node size is probably fine, but I'm hedging with 'probably', just in case...
15:09:02 <nibalizer> ok
15:09:05 <SotK> s/5/6/
15:09:20 <nibalizer> haha yes we have 119 workers
15:09:34 <nibalizer> all who amusingly have  --worker-count 5 in their proccess name
15:09:37 <Zara> xD
15:09:50 <Zara> i wonder if that's a clue?
15:10:08 <Zara> (hopefully can fix, otherwise we'll have to kill them manually again, but that's going to get annoying. every three months, a scheduled storyboard cleanup... yeahhhh... :/)
15:10:14 <nibalizer> this comes back to having a dev sevrer would be great because then Zara and SotK can have shell and poke
15:10:39 <SotK> I've figured out why they haven't been dying properly now, next I'm trying to figure out a way to fix it
15:10:44 <nibalizer> is it safe for me to just kill these with prejudice?
15:10:49 <Zara> (yeah, SotK has some kind of setup to test it, but my understanding is that it was a pest to set up and slow)
15:11:13 <SotK> nibalizer: yeah, feel free to kill them all, just restart the storyboard-workers service afterwards
15:11:13 <Zara> nibalizer: I'm happy with that, SotK is it useful for you testing for them to stay alive for a bit?
15:11:18 <Zara> ah, snap
15:11:20 <Zara> nvm
15:11:33 <Zara> excellent, thank you
15:11:51 <SotK> also, thanks :)
15:12:25 <nibalizer> cool
15:12:31 <Zara> #action nibalizer to kill all the excess processes with extreme prejudice \o/
15:12:43 <Zara> yayyy
15:12:49 <Zara> any other urgent items from anyone?
15:13:11 <SotK> _o_
15:13:21 <Zara> guess if they were urgent, we'd know about them :)
15:13:26 <Zara> #topic In-progress work
15:13:47 <Zara> #info SotK has been working on worklists and boards some more (automatic boards!)
15:13:56 <SotK> oh yeah
15:14:11 <Zara> I was going to point to them, but webclient is behind because of aforementioned problems.
15:14:20 <SotK> its possible to edit worklists from the board view now (or will be when the webclient updates)
15:14:33 <SotK> this lets you make a board, then make its lanes be automatically populated
15:14:40 <Zara> super secret automatic boards in master, coming soon to a storyboard.openstack.org near you
15:14:55 <Zara> I'll put a link to an example on the etherpad when I can make the example :)
15:14:56 <SotK> such lanes won't be manually usable though (no dragging)
15:15:30 <Zara> heh, I have no idea how someone would even begin to make the rules for that.
15:16:18 <Zara> well, I guess the act of removing something from a lane would have to add a very specific rule to it... anyway, /speculation
15:16:37 <Zara> #info SotK has been investigating the worker mystery
15:16:57 <Zara> I found that at 2amish, so that's been all today...
15:17:15 <SotK> the workers sometimes get stuck here when stopping: https://git.openstack.org/cgit/openstack-infra/storyboard/tree/storyboard/plugin/event_worker.py#n156
15:18:00 <SotK> I don't really know why they get stuck there, seems racy
15:18:09 <Zara> still seems like you're making progress :)
15:18:24 <Zara> last time I think the whole thing was a total mystery, this time you know where they're stuck
15:19:19 <SotK> yeah, should get somewhere this time
15:19:39 <Zara> :) do you have an idea of what possible reasons to investigate?
15:19:46 <SotK> (I've seem to have fixed it by not cancelling that timer, but that seems suboptimal)
15:20:00 <Zara> I can't help much but can at least be a person to think out loud to
15:21:02 <SotK> beyond "figure out what would cause threading.Timer().cancel to hang", not yet
15:21:34 <Zara> heh. :) if you're blocked, I can listen and/or find pictures of armadillos.
15:22:43 <Zara> which should reduce stress, if nothing else
15:22:56 <Zara> though I'm not a medical professional so don't cite that anywhere
15:23:19 <Zara> as for me... I've mainly been reviewing this week
15:23:43 <Zara> #info Zara has been reviewing, and is now preparing storyboard status report for summit
15:23:50 <Zara> and anything else that needs preparing...
15:24:23 <Zara> I'm hoping to wait 'til after newer webclient builds, because otherwise I'll just have to rewrite
15:24:55 <Zara> so maybe 'weird misc' is a better title for my inprogress work
15:25:07 <Zara> business as usual.
15:26:06 <Zara> updated a couple of tiny old patches, replied to some stories, updated an ehterpad, need to update it more... eh
15:26:21 <Zara> am I missing anything?
15:26:38 * SotK doesn't think so
15:26:48 <Zara> there was a *lot* of review :)
15:26:58 <Zara> okay, I also put betherly's tutorial on the agenda!
15:27:23 <Zara> betherly, are you about? have anything to say about it? is it still on fire?
15:27:24 <betherly> awesome
15:27:37 <betherly> its happening for sure. i hit a pretty big road block with d3
15:28:19 <Zara> :) is the roadblock the kind of thing we'd benefit from you documenting?
15:28:22 <betherly> had to scrap quite a lot of the work to go with a new approach but checking out some new examples to check im going in the right direction now
15:28:43 <betherly> Zara: maybe. i will try and put some stuff together :)
15:28:57 <Zara> cool, I'll pester you for a note on the story if so, otherwise dw about it. :P
15:29:29 <betherly> ok awesome. it will probably be after the summit now as trying to get a load of stuff ready for it
15:29:29 <Zara> I'm not sure of a better place to document it round these parts, but your pain might save us some time in the future xD
15:29:48 <betherly> just sad i wont have more to show for my work next week :( was hoping to have something cool to show y'all
15:30:18 <Zara> aw, np, the 'don't do this it will end in flames' is actually a cool thing to show
15:30:27 <Zara> even if it doesn't feel like it
15:30:43 <betherly> ok cool i will do that :)
15:30:52 <Zara> thank you again for helping out with this
15:31:14 <Zara> and hopefully we can talk more at summit :D
15:31:24 * SotK hopes so!
15:31:41 <betherly> awesome
15:32:37 <Zara> #info betherly trying new approach for tutorial after hitting roadblock, Zara will pester her to document roadblock
15:32:41 <Zara> hopefully that's accurate :P
15:32:50 <betherly> yep thats accurate
15:32:52 <betherly> thanks Zara
15:33:11 <Zara> :) no discussion topics on agenda, soooo...
15:33:17 <Zara> #topic Open Discussion
15:33:33 <Zara> (and you're welcome, I haven't really done anything, but you're welcome anyway :P)
15:33:44 <Zara> so... well, it's summit soon
15:33:50 <Zara> that seems like a smart thing to discuss
15:33:51 <Zara> openly
15:34:15 <Zara> we've had questions about whether there are storyboard sessions at the summit
15:34:35 <Zara> so, there's the community task-tracking session, but other than that, it's pretty ad-hoc
15:35:28 <Zara> the vague plan in my head is 'coordinate schedules with other active storyboard developers, then if someone wants to talk, let them know where we are'
15:36:17 <SotK> seems like a sensible plan
15:36:31 <Zara> the 'where' might well be 'comfy seat by snack table with decent wifi'
15:36:34 <betherly> that sounds good
15:36:59 <betherly> im going to be mental busy especially the second half of the week bouncing around ironic/horizon/ux sessions
15:37:19 <Zara> okay. my monday is mostly (maybe completely) free atm.
15:37:31 <betherly> but if theres something happening or you want to meet up ping me and i will pick it up
15:37:40 <Zara> (in terms of sessions I absolutely should be attending, anyway)
15:38:32 <betherly> monday is pretty good for me atm too so lets make a plan to meet up then
15:38:42 <Zara> so monday is the best chance for storyboard pool party-- er I mean, storyboard working session
15:38:48 <betherly> and sunday night i will see you at the WOO event Zara?
15:39:07 <betherly> i land at 5:30pm though following a 30 hr flight though so dont expect much sense out of me ;)
15:39:54 <Zara> betherly: ack, I should be there, fine with leaving early so worry not
15:40:15 <Zara> I haven't even looked at weds thru fri in detail yet
15:40:16 <betherly> cool thanks Zara :)
15:40:25 <betherly> i need to do some serious planning
15:41:05 * SotK too
15:41:14 <Zara> tues has a few sessions, but I should still have a fair bit of time then in pinch.
15:42:18 <Zara> oh, also, based on last time, a quirk of storyboard is that the most relevant sessions tend to be tues/weds
15:42:59 <Zara> because of the cross-project and infra things. we actually have more flexibility at the end of the week, because we don't clash with, say, nova
15:43:03 <Zara> and so on
15:44:10 <Zara> though I say that without looking at the schedule yet
15:46:44 <Zara> just yeah, I would assume much of monday is going to be getting to grips with the venue etc, so good for less formal meeting, tues and weds we'll be the least flexible, and the end of the week will be better for more focused  hacking.
15:47:03 <Zara> but that's a very vague forecast and doesn't take into acct everyone being sleepy by the end of the week
15:52:08 <Zara> any other things to discuss?
15:52:11 <SotK> _o_
15:52:21 <Zara> I *should* have worked out the schedule a bit more before this, so sorry about that
15:57:01 <Zara> so, some sort of tentative storyboard monday meet planned, task tracking session weds morning, other storyboard summit things tbc
15:57:50 <SotK> seems good to me
15:59:16 <Zara> we'll keep people posted in #storyboard, the one-stop-shop for all your storyboard needs!
15:59:35 <Zara> #endmeeting