22:04:19 <jeblair> #startmeeting zuul
22:04:20 <openstack> Meeting started Mon Aug 28 22:04:19 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:04:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:04:23 <openstack> The meeting name has been set to 'zuul'
22:04:39 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:04:51 <jeblair> #link actual agenda https://etherpad.openstack.org/p/zuulv3-pre-ptg
22:05:06 <clarkb> hello
22:05:16 <jeblair> #topic tarball/publish jobs
22:05:32 <jeblair> pabelanger: i think these are done, yeah?
22:05:38 <pabelanger> o/
22:05:41 <pabelanger> yes, done
22:05:54 <jeblair> thanks!
22:05:59 <jeblair> #topic devstack jobs
22:06:06 <jeblair> #link https://etherpad.openstack.org/p/AIFz4wRKQm
22:06:14 <jeblair> i sent an email about this
22:06:31 <jeblair> i got generally good response, so i think our current plan is two forks
22:06:54 <jeblair> 1) devstack-legacy job which just tries to run the current devstack-gate script with few changes
22:07:14 <jeblair> 2) (new) devstack job that is v3 native
22:07:20 <mordred> ++
22:07:22 <fungi> oh, yes i read it and am also generally in favor of the compromise described for now
22:07:36 <fungi> sorry, still trying to catch up on e-mail backlog from last week
22:07:52 <mordred> fungi: you need a post vacation vacation to deal with the fallout from the vacation
22:08:06 <pabelanger> I'm happy with the 2 jobs approach
22:08:08 <fungi> this is my post vacation vacation
22:08:12 <jeblair> i feel fairly sure we can have devstack-legacy going very soon now.  i'm focusing on that.
22:09:02 <jeblair> i'm hoping we can have the new devstack job (i call it devstack-ng in my head, but really its name should just be devstack) ready (or mostly ready) by ptg as well, so we have something to show, even if it's not fully built-out and used.
22:09:47 <pabelanger> great
22:10:10 <jeblair> dmsimard is working on the network overlay stuff
22:10:25 <fungi> yeah, getting _a_ devstack job going under the new model is likely a lower bar than translating all the various devstack jobs we have into the new model
22:10:26 <jeblair> which will be used in both jobs (in one form or another)
22:10:39 <jeblair> fungi: yeah, and lets us build up in complexity
22:10:48 <dmsimard> yup, tackling one issue after the next
22:10:54 <mordred> fungi: ++
22:11:36 <clarkb> dmsimard: I actually find it weird we put libffi-devel in centos-minimal fwiw and don't see where we put it in ubuntu
22:11:39 <jeblair> my best guess for devstack-legacy (which is our minimum at this point) is mid-week.
22:11:43 <dmsimard> I thought the current patchset was good to go but there's still an issue -- I might need a hand with that one or it's possible it might be completely unrelated
22:11:46 <clarkb> dmsimard: I think things may work now because cryptography doesn't dep on that? I dunno
22:12:19 <fungi> pyca/cryptography doesn't need it if you install from a wheel, i believe
22:12:31 <fungi> but if you install from sdist then it does
22:12:35 <mordred> right. and we have a wheel mirror that builds wheels of cryptography
22:12:43 <dmsimard> clarkb: you typically need both libffi and openssl devel headers to compile crypto yeah.. but then again, we install from wheels so we don't compile ?
22:12:54 <dmsimard> yeah what mordred said
22:13:07 <clarkb> oh ya thats probably it. So I don't think we can merge your chagne safely yet
22:13:22 <dmsimard> which one, paramiko ?
22:13:28 <clarkb> yes unpin paramiko
22:13:51 <dmsimard> ok it's a bit unrelated to the network overlay stuff, we can discuss it off meeting
22:14:02 <clarkb> ok
22:14:31 <jeblair> any other devstack stuff?
22:14:32 <SpamapS> o/
22:14:41 * SpamapS had meatspace troubles, here now
22:14:48 * mordred hands SpamapS a plate of chicken
22:15:02 <jeblair> is chicken meat?
22:15:02 <SpamapS> omnom devstack chicken
22:15:08 <jeblair> #topic jobs that use special slaves
22:15:16 <jeblair> #link https://etherpad.openstack.org/p/zuulv3-special-jobs
22:15:23 <jeblair> this is a new addition since last week
22:15:32 <jeblair> but mordred has been tackling it
22:15:38 <mordred> so I went through and made a list
22:15:44 <mordred> of all the things that use "special" slaves
22:16:03 <mordred> I've been working on base jobs for the proposal ones which I think are in the process of landing right now
22:16:58 <pabelanger> ya, I have some changes up for AFS publishing, so that overlaps with wheel mirrors for sure
22:17:04 <mordred> it does indeed
22:17:14 <jeblair> since we haven't merged the nodepool static node patches, we're fairly committed to avoiding use of those for the initial v3 rollout at least; thus all of these need to be ported to use non-special nodes (which is ideal anyway)
22:17:15 <pabelanger> sonce once we land the afs bit, wheel mirror should fall into place
22:17:22 <mordred> pabelanger: I was goign to see if I could sell you on taking the AFS related ones from the list since you're already AFS-ing
22:17:37 <pabelanger> mordred: ya, should be okay with that
22:17:40 * rbergeron whistles and swings by
22:17:42 <Shrews> jeblair: should we work on landing that?
22:17:46 <mordred> pabelanger: wheel mirror is, unfortunately, a bit of a special case- but we can chat about it post-meeting
22:17:56 <pabelanger> mordred: ack
22:18:14 <fungi> i see tarball signing jobs aren't in that list... i guess because we already have that working with single-use nodes in zuul v3?
22:18:30 <mordred> Shrews: I think there's still a conceptual issue we need to address with using static nodes for these jobs that we should discuss at the ptg- but I don't think it'll accelarate these jobs for now
22:18:36 <jeblair> Shrews: i'd rather wait until after ptg; i don't think we have time to build up a solution around it, and it's not actually what we want for these jobs anyway.
22:18:41 <pabelanger> fungi: ya, we do that in bubblewrap on the executor today
22:18:43 <mordred> fungi: yup. tarball signing is actually done
22:18:52 <Shrews> mordred: jeblair: ack
22:18:57 <mordred> fungi: (it's in the list, just struckthrough)
22:18:57 <fungi> cool so it's one that _would_ have been in the list but was tackled early
22:19:01 <mordred> yup
22:19:01 <jeblair> (basically, if using dynamic nodes for this fails, that increases the priority of the static patches)
22:19:14 <fungi> oh, yep nevermind, it was under wheel mirror building which is why i missed it
22:20:50 <jeblair> mordred: other than handing afs stuff off to pabelanger, is the rest amenable to other folks pitching in, or do you think you'll knock 'em out yourself?
22:21:37 <mordred> jeblair: both are true - the common proposal stack is basically done (remaining piece is part of migration script, since it's about registering job variants)
22:21:54 <clarkb> I feel like I am missing something, were we considering keeping these jobs on static purpose built instances?
22:22:13 <fungi> sounds like the preference is to avoid doing that
22:22:16 <dmsimard> I'll keep hacking on the devstack stuff, there are other things I saw I could help with (i.e, create zuul stream unit tests) but I guess devstack is the priority
22:22:19 <jeblair> clarkb: i consider it a backup plan that's still a backup plan :)
22:22:26 <clarkb> gotcha
22:22:39 <mordred> the release jobs and the special proposal jobs are all independent, so could certainly benefit from pile-on if people feel like helping
22:23:11 <fungi> the reference to these jobs being a "special case" is that they're not particularly generalizable and may need some special support executor side for some things
22:23:11 <mordred> but are all largely going to be about re-using the "copy the script and execute it" logic from the proposal job for now- so I expect to hammer them out quickly
22:23:39 <mordred> oh - no - just they're not the same as using the propose_update.sh script - so they need to be looked at individually
22:24:06 <jeblair> and the approach mordred's taking is basically to keep using the scripts and not ansible them for now; defer that until after ptg.
22:24:14 <mordred> so "special case" here means they're not being handled as part of a larger more systemic thing -like the things that publish to AFS
22:24:16 <fungi> though it sounds like all the executor-side support we think we might need is in place at this point?
22:24:21 <dmsimard> mordred: I did find a silly ansible bug when I was copying scripts verbatim for d-g migration, I'm expecting it to probably show up when translating the jobs from JJB: https://github.com/ansible/ansible/issues/28674
22:24:44 <dmsimard> So, for that reason, it'd probably be best to effectively run a task that runs a script, and not embed the actual script in a shell task
22:24:48 <jeblair> fungi: yeah, mostly depends on secrets and i think that's workable now
22:25:12 <mordred> dmsimard: yes - and that's what we're doing - totally agreee
22:25:27 <fungi> oh awesome, so we can't use apostrophes in job comments? :/
22:26:05 <fungi> (and presumably a host of other stuff which might confuse the parser)
22:26:22 <dmsimard> fungi: yeah, exactly.. there's too many ways this could go wrong
22:26:31 <dmsimard> running things as-is the the safest way
22:26:31 <mordred> yah - for the current 2.5 jjb translation we put all the script blocks into external files and execute them that way
22:26:50 <mordred> which I expect to do for the actual migration, since, well, we know it works today :)
22:26:53 <jeblair> #topic migration script
22:27:01 <jeblair> (topic change since we're there anyway)
22:27:16 <mordred> I started this a couple of weeks ago then shelved it to work on jobs
22:27:41 <mordred> my current plan is to return to hacking on it as soon as I get those proposal/release jobs knocked out or handed off
22:27:49 <jeblair> yeah, i still think this isn't ripe until that + devstack-legacy are in place
22:28:00 <mordred> yah
22:28:25 <jeblair> anything else on the subject?
22:28:50 <jeblair> #topic migration docs
22:29:00 <jeblair> i haven't worked on this at all since last week
22:29:40 <jeblair> hopefully if devstack-legacy starts actually running for more than 2m before bailing, i'll have some intervals to add bits here
22:29:41 <mordred> Shrews did some patches
22:29:52 <jeblair> ya, there's a walkthrough there now
22:30:48 <jeblair> i don't think we can flip the switch until we have a document that says "hey, if you want your job non-voting, forget all that stuff from zuulv2, you want a job variant".  that's the stuff that still needs writing.
22:31:15 <pabelanger> I'll make some time this week to look at the docs also
22:31:49 <jeblair> #topic meeting next week (bonus topic!)
22:32:05 <jeblair> next monday is a us holiday
22:32:06 <fungi> bonus round!
22:32:08 <mordred> from previous topic: https://review.openstack.org/#/c/498111 ... if someone wants to +A that one in
22:32:18 <dmsimard> next monday is also a canada holiday :)
22:32:23 <dmsimard> (for pabelanger and I)
22:32:40 <mordred> it's a canada holiday for all of us
22:32:40 <pabelanger> pfft, days off?
22:33:06 <jeblair> so maybe let's skip the meeting next week; should we try to resync on tuesday though?
22:33:22 <fungi> i'm up for observing a canadian holiday next week
22:33:41 <dmsimard> can have a sub-topic in the -infra meeting
22:34:07 <mordred> I'm up for more frequent than usual check ins over the next two weeks as needed
22:34:23 <jeblair> dmsimard: yeah, that sounds like a great plan for 1 week before the ptg regardless.
22:34:52 <dmsimard> mordred: yeah we're informally keeping in sync with each other day to day
22:34:54 <rbergeron> there are countries other than canada? i refuse to believe this
22:35:05 <fungi> dmsimard: now you have to convince clarkb when you want things in the infra meeting agenda
22:35:08 <jeblair> rbergeron: it's a temporary setback
22:35:17 <fungi> bribing me no longer works ;)
22:35:28 <dmsimard> but imo let's have a zuul v3 topic in -infra meeting if it's okay with fungi/clarkb :p
22:35:35 <rbergeron> i have bonus bonus next week info if anyone is interested :)
22:35:52 <dmsimard> rbergeron: you show up just to give us bonuses? I like
22:36:05 <Shrews> rbergeron: free hotdogs?
22:36:11 <jeblair> #topic next week bonus info (bonus bonus topic!)
22:36:12 <rbergeron> Shrews: you know it :)
22:36:37 <rbergeron> ah -- i was just going to say, I don't want to interrupt jim's agenda until "bonus topics time," but here we are
22:37:15 <rbergeron> so: AnsibleFest is next week in SFO -- I know a few of y'all are coming for one reason or another, and though we won't have organized Zuul Things like we have in the past..
22:37:26 <rbergeron> (turns out y'all are shipping things, and stuff, the following week) --
22:37:34 <jeblair> #agreed skip sept 4 zuul meeting; catch up on sept 5 infra meeting
22:37:56 <jeblair> i was planning on stopping by
22:38:04 <rbergeron> for thos e of you who are coming and want codes for ansiblefest, or to come to the contributor things, or to come to the cocktail party on the evening of Sept. 6, please send me a mail or ping me like, right now.
22:38:11 <fungi> it's a smidge out of the way for me, but sounds fun!
22:38:39 <clarkb> fungi: roadtrip!
22:38:44 <rbergeron> I will casually mention that if uyou come you will get to witness either the awesomeness or hilarity of robyn interviewing jim whitehurst as a keynote, and he did indicate he would love to come to the cocktail party and meet our fine contributors.
22:39:05 <dmsimard> Wish I could've gone :(
22:39:13 <rbergeron> And I hope to buy you all really expensive cocktails in denver after i arrie post open-source-summit in los angeles.
22:39:23 <jamielennox> one of the worst hangovers i've ever had from that cocktail part
22:39:26 <fungi> you just made me look up who that is
22:39:48 <dmsimard> fungi: he's some random guy :)
22:39:56 <rbergeron> dmsimard: sorry :\ i wish i had been on less fire with "announcements" and "work" and shit to tend more to this fest / contributor summit, but :\
22:40:29 <rbergeron> jeblair: it is in SFO (marriott marquis downtown), i the event you want to swing by and hug pabelanger or maria bracho in person, along the with the everyone elses.
22:40:30 <dmsimard> rbergeron: no worries, bring the next north america ansiblefest in Montreal :D
22:41:03 * rbergeron motes that some openstack foundation humans of awesome seem to be at some other event also in SFO those days and are beinb invited to cocktail things
22:41:10 <jeblair> rbergeron: oooh that's the one with the deathstar windows in the view bar... :)
22:41:12 <fungi> hrm, rh ceo for 9 years? i really have a _terrible_ memory for names
22:41:21 <rbergeron> and, well, I guess that ends my "hi, i'm robyn, and i'm here to tell you about the aailability of alcohol" portion of this meeting agenda :)
22:41:58 <fungi> rbergeron: oh, is ansiblefest happening at the same time as opendev?
22:42:24 <rbergeron> fungi: he's a super nice guy. And super smart -- he came to the ansible office to chat last week on eclipse day (aka my boss's birthday), and we talked about kanban and when he was COO at Delta and loving linux and nerds and also really heavy crazy shiz. :)
22:42:28 <jeblair> http://cache.marriott.com/propertyimages/s/sfodt/sfodt_main03_r.jpg
22:42:28 <rbergeron> fungi: yassss.
22:42:37 <fungi> whups
22:43:02 <rbergeron> jeblair: omg they really are death star windows
22:43:23 <jeblair> the effect is awesome at night :)
22:43:37 <rbergeron> fungi: no worries, it's less awkwards than the PTG at the same time as the linux foundations open source summit and my sads being the cloud track chair for that knowing all the humans i love are away getting things done :)
22:43:48 <jeblair> #topic devstack-gate and git cache (bonus bonus bonus topic!)
22:43:59 <fungi> lightning round!
22:44:04 <jeblair> since we have a few minutes -- i wanted to mention this real quick...
22:44:04 <rbergeron> fungi: good news is... we have a cocktail party the night before and can invite all the humans we'd like :)
22:44:48 <jeblair> the devstack-legacy job has many more git repos than a typical zuulv3 job will have (clarkb has whittled down the projects list a lot, but not completely)
22:45:14 <jeblair> this means we may need to push a lot more data to the nodes than we would have expected
22:45:19 <pabelanger> I actually had a question about this too, how soon are we stopping cache-devstack DIB process? Things like cirros images for example.
22:45:37 <jeblair> so i'm going to look at using the git repo cache for that (which we otherwise don't use)
22:46:02 <jeblair> but in doing so, i'd like to ask questions like what pabelanger just asked --
22:46:11 <fungi> so the executor would push the prepared refs into the cached local copies on the nodes, i guess?
22:46:16 <jeblair> fungi: yep
22:46:27 <fungi> seems sane enough on the surface anyway
22:46:30 <jeblair> basically -- if we do this, how long do we want to do it for?  how important is removing the git repo cache from images, etc.
22:46:40 <jeblair> i'm going to add this as an infra meeting topic
22:46:59 <jeblair> because this is, at base, an openstack infra policy question, so that's a better venue
22:47:04 <mordred> it's a great topic
22:47:20 <fungi> one thing i think we're missing still (and can always discuss more tomorrow) is performance metrics to see how much it hurts to do that without the local cache
22:47:28 <jeblair> but wanted to mention it here to make sure folks know about that, and i'll probably be pushing up patches soon which may lead to confusion.
22:47:38 <pabelanger> ++
22:47:48 <jeblair> fungi: yeah.  infra-cloud complicates this too.
22:48:02 <fungi> i also think we haven't clearly enumerated what things we want to stop caching, and what benefits we believe we're get from doing that
22:48:15 <jeblair> since we now have a cloud where nodes underperform my home dsl in their networking.
22:48:52 <jeblair> that's *usa* dsl, just to be clear.  so, like, nothin'.
22:48:56 * fungi wants to request we upgrade that isdn line to dual-channel
22:49:20 <jeblair> anyway, think about that and we'll talk tomorrow :)
22:49:25 <mordred> ++
22:49:29 <pabelanger> nodepool-builder uploads are working very well on rackspace (nb04). We can bring new images online in about 30mins (80GB total) now
22:49:37 <pabelanger> nb03, little longer
22:49:51 <fungi> nb03 is the one in vh?
22:49:56 <pabelanger> ya
22:50:15 <clarkb> fungi: I've also got a ptg idea on the etherpad to talk about what we host on our mirrors
22:50:15 <fungi> do we think that's due to network, disk, or some other bottleneck?
22:50:20 <clarkb> these topics are probably tightly related
22:50:25 <mordred> clarkb: ++
22:50:28 <pabelanger> fungi: network, we only have 100Mbps
22:50:30 <fungi> agreed
22:50:37 <pabelanger> rackspace is 400Mbps (IIRC(
22:50:38 <fungi> on the ptg topic idea
22:51:49 <jeblair> any more bonus bonus bonus bonus topics?
22:52:40 <jeblair> thanks all!
22:52:42 <jeblair> #endmeeting