19:02:46 #startmeeting infra 19:02:47 Meeting started Tue Apr 25 19:02:46 2017 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:47 presents? 19:02:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:51 The meeting name has been set to 'infra' 19:02:55 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:57 #topic Announcements 19:03:04 #info Let fungi know if you hope to attend the PTG in Denver this September so he can get a rough head count 19:03:12 as always, feel free to hit me up with announcements you want included in future meetings 19:03:19 #topic Actions from last meeting 19:03:26 #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-04-18-19.02.html Minutes from last meeting 19:03:34 #action fungi put forth proposal to flatten git namespaces 19:03:37 still drafting a spec locally based on ttx's brainstorming etherpad 19:03:40 #link https://etherpad.openstack.org/p/repo-name-shortening-impact Impact of shortening repository names 19:03:42 this has finally bubbled to the top of my to-do backlog at least 19:03:50 #action pabelanger Open an Ubuntu SRU for bug 1251495 19:03:52 bug 1251495 in mailman (Ubuntu Trusty) "Lists with topics enabled can throw unexpected keyword argument 'Delete' exception." [High,Triaged] https://launchpad.net/bugs/1251495 19:03:52 i checked the bug a few minutes ago so assuming this is still outstanding 19:04:03 still ongoing, sadly 19:04:10 #action clarkb to add citycloud to nodepool 19:04:13 #link https://review.openstack.org/458621 Update citycloud to non demo accounts 19:04:15 #link https://review.openstack.org/458622 Add citycloud to nodepool 19:04:18 these are presumably wip until we know what regions we'll be using and confirm appropriate quota 19:04:50 yup 19:04:54 cool 19:05:00 I don't want to bug them but maybe I should 19:05:02 #topic Specs approval 19:05:07 we don't seem to have anything new up this week 19:05:12 #topic Priority Efforts 19:05:19 nothing called out specifically here 19:05:28 #topic New propose job for OpenStack Ansible (hwoarang, odyssey4me) 19:05:32 #link https://review.openstack.org/453130 Add 'propose' job for openstack-ansible-tests 19:05:36 per the agenda, "Need to either agree to do it or reject it and proceed with manual syncing instead" 19:05:39 concerns raised are the usual outlined in our reviewing documentation 19:05:45 #link http://git.openstack.org/cgit/openstack-infra/project-config/tree/REVIEWING.rst?id=291daea#n51 Reviewing project-config: Proposal jobs 19:05:51 the stated use case seems similar to this change which we approved a year ago: 19:05:55 #link https://review.openstack.org/301375 Implement periodic job to update Puppet OpenStack constraints 19:06:35 It might be worth reaching out to openstack puppet team, since they last did this and quickly revert it 19:06:51 well, they reverted the delorean proposal job, right? 19:06:54 IIRC, they got tired of +3 automated patches 19:07:14 not delorean, but job promotion things 19:07:29 ahh, i assumed "dlrn" was delorean 19:07:37 something else then, i guess 19:08:04 But, no objection from me, if they want to give it a go 19:08:25 the propose-puppet-openstack-constraints job seems to still be in place anyway 19:08:56 yeah, still in the periodic pipeline for openstack/puppet-openstack-integration 19:09:23 odd, maybe I'm thinking of some other one. 19:09:30 I'll dive into in more later 19:09:39 they removed their dlrn promotion proposal job 19:09:49 saw that in our git history when researching this current request 19:09:50 k, that must have been it 19:10:48 ran successfully today for example: 19:10:49 #link http://logs.openstack.org/periodic/propose-puppet-openstack-constraints/0595ba5/console.html 19:12:19 hwoarang and odyssey4me don't seem to be present to discuss this, but does anyone disagree with the basic idea of 453130 anyway? 19:12:31 I am around 19:12:35 ahh, cool 19:13:33 I am happy either way. A job is convenient but we can also do it with a script 19:13:54 could we maybe make them a python module and consume them that way? 19:14:26 some things like the shared bindep.txt file may be hard to cover that way 19:15:22 i guess it's a question of how much of the things which change frequently can be covered by a shared dependency, and whether what's left changes often enough to benefit from automated consistency 19:15:47 They don't change often to be fair 19:15:57 Since they are only used for manual testings 19:16:07 Not for the gates 19:16:17 Apart from bindep.txt 19:16:32 personally, we've had the devstack plugin doc proposal job for a while ... workflow seems to be fine, even though yes it could be done other ways. the bindep bit seems most convincing to me 19:16:39 Which should change once in a blue moon 19:16:51 it's always possible to make a different builder macro to use a shared bindep.txt from some central repository you maintain too 19:17:22 and if zuul-cloner's checking that out, you can reliably use depends-on relationships when changing it too 19:17:49 fungi: i feel like the bindep in the source tree should be the one ... otherwise aren't we making it quite hard for externals? 19:17:49 But we also need to consider local testing 19:17:51 cd 19:18:13 We need to have a current bindep in place without depending on another repo 19:18:14 we had a similar discussion about the ~90 puppet module repos the infra team maintains, and decided to just stick with a shared gem as a dependency for the things we could cover that way and manually handle everything else on the rare occasion we need to change it 19:19:35 right thats why I'm wondering if something like pypi package would be good 19:19:38 gem, or a requirements.txt addition seems a bit different, since that's already a "go and grab it" type situation 19:19:56 maybe if bindep had a "include from http" method ... 19:20:36 i'm not opposed to bindep including remote references, though that is a fair amount of added complexity and error handling 19:20:48 True 19:21:33 ansible should be learning about bindep shortly: https://github.com/ansible/ansible/pull/22159 19:21:47 so, it wouldn't be hard to write a playbook to fetch bindep.txt file first 19:22:15 so from my perspective, i'm not opposed to the requested job... but it is very easy to rely on automated code duplication for things where having actual shared dependencies is more appropriate, so have to make sure to avoid that temptation 19:23:02 it doesn't seem like it would be proposing new changes particularly often, so likely don't need to worry about the reviewer fatigue downside anyway 19:24:29 Yes i want to make sure all changes are in before the job is in place so we know only have one big round of reviews and that's it 19:25:03 Sorry typing from phone 19:26:39 current list of files it's syncing are run_tests.sh, bindep.txt and Vagrantfile 19:26:49 Yes 19:26:55 which presumably are all similarly hard to cover with a python dep 19:27:47 ya, they don't slot in directly 19:28:06 i'm inclined to approve it once the issue AJaeger has identified is corrected 19:28:22 AJaeger: any opinion on this since you've been the most engaged reviewer on that change? 19:28:28 so what we want to solve here is the vagrant bit 19:28:48 we expect users and devs to simply do 'vagrant up' and start the testsuite 19:28:50 fungi: I'm fine with taking it 19:29:10 i am also happy, per comment; modulo AJaeger's review 19:29:13 okay, cool. i feel like we've done some thorough diligence discussing it 19:29:15 for what we need Vagrantfile and run_tests.sh in the repo. Otherwise the user will have to fetch these files from omewhere else 19:29:23 *that 19:30:18 #agreed The use case for the "propose-openstack-ansible-tests-scripts" job put forth in change 453130 is sufficiently justified 19:30:42 thanks for bringing this one to the meeting hwoarang! 19:31:02 #topic What to do about debs for vhd-util and bubblewrap (mordred) 19:31:04 thank you for all the discussion around it. 19:31:08 per the agenda, "PPAs on Launchpad seem to be sad, as does the backports process currently" 19:31:29 yah. 19:31:50 so - according to SpamapS there is a dearth of humans processing the backports queue 19:31:53 i don't know that i have what it takes to make them happy. 19:32:05 * jeblair tries balloon animals 19:32:07 this is the normal way we'd consume bubblewrap on xenial 19:32:26 there's of course the filthy option of building these and sticking them in a directory on tarballs.o.o by hand. not that i'm advocating for such a thing 19:32:48 but there's probably an acceptable middle ground between ubuntu ppa and that 19:33:01 right so my idea for middle ground was to use nix 19:33:18 we might have to make a nix package but then it would theoretically be useable on centos/fedora/ubuntu/gentoo/whatever 19:33:41 fungi: not sure that's all that filthy ... since all the building is logged and repeatable. afs kernel modules for example 19:34:04 mordred: I have gotten some feedback on backports 19:34:13 other options include hosting our own .deb repo (apparently we're closer to be able to do that now after all the deb packaging work?). and another option is switch to another os. 19:34:20 ianw: well, ultimately i think we want some build automation to exist for anything we serve from tarballs.o.o 19:34:20 Apparently there's more or less 1 person doing backports, and he went on vacation last week. 19:34:55 using nix also potentially gives us a good answer for when people want $reallynewthing like libvirt or ovs in $job 19:35:26 Iain Lane has also basically suggested that the process is too cumbersome and they'd like to make backports more developer-self-service 19:36:14 it's not as if rerunning apt-ftparchive every time we upload a package into the directory is all that hard to implement if the desire is to host a deb package repository 19:36:29 clarkb: help me out. nix packages are source only? Or they build in chroots for all abi's? 19:36:59 fungi: I agree.. hosting a small collection of no-change backports is quite simple and many many many shops do it. 19:37:37 SpamapS: aiui its both? 19:38:03 SpamapS: you can bootstrap locally by compiling from the ground up or you if you can use published packages 19:38:13 building on deb package work will require some work, since it was debian-jessie based. However, not the end of the world 19:38:28 and since nix installs are self contained if you grab published packages it just grabs all the published deps too and puts them in the install root 19:38:42 SpamapS: yeah, my script for it looks like this (makes secure-apt perfectly happy too wen my openpgp key is installed): http://paste.openstack.org/show/607916/ 19:38:55 and in theory this will all just work except for the kernel because well that is harder to replace on the fly 19:39:14 clarkb: yeah so, it is basically adoptiong nixos :) 19:39:23 SpamapS: no 19:39:26 just not the kernel :) 19:39:28 no 19:39:41 everything nix would be in its own curated corner that you'd opt into as the user 19:39:53 adopting nixos implies a lot more changes 19:39:58 in /opt/into/nix! 19:40:01 especially around image building and config managment 19:40:07 Yeah, you're adopting nixos, for bubblewrap. :) 19:40:16 I like it 19:40:19 don't get me wrong at all! 19:40:22 yes but not necessarily for zuul if that makes sense 19:40:26 It's a container approach really. :) 19:40:34 yes its similar to snaps 19:40:38 or the flatpak stuff 19:40:47 which uses bubblewrap ;) 19:40:50 but its been around longer and seems to be much richer 19:40:54 or rpath/conary, or... 19:40:55 * SpamapS just went circular 19:41:00 right because its not actually doing process isolation 19:41:10 its just telling your linker and path and all that where to load things into memory 19:41:14 once thats done its up to you 19:42:16 its just a fancy way of putting bits on the filesystem so that when added to your path they just work 19:42:30 like software collections? 19:43:00 ianw: ya except its on a per user basis and it dedups and does fancy things 19:44:20 anyways, I was just thinking that maybe instead of building $package x distro x releases we could make packages in a system that we'd set up once 19:44:55 snaps and flatpaks are other options here and they come with different garuntees and drawbacks 19:46:07 so i guess we need short-term and long-term solutions... the vhd-util package is something we need a means of updating on our own and may not be able to wait for extensive package build automation to be implemented (be that nix or whatever) 19:46:31 and getting a bubblewrap package is a blocker for zuul v3 rollout 19:46:36 what's the status of our .deb build automation? 19:47:36 Whatever does the apt mirror building could just as easily process uploads from an incoming dir, yes? 19:47:56 (not into the same mirror, but into the same afs) ? 19:47:58 i mean, i have no experience with any of those tools in production/dev-ops situations ... so not sure i'd have anything useful to add. TBH, the vhd-util stuff, a small periodic build and push to a tarballs repo seems fairly simple and well tested 19:47:58 (i heard things happened for the openstack debian work, but i don't know what) 19:48:13 oh right there is already some deb build automation 19:48:14 yes, openstack debian is still a thing, but currently debian-jessie based 19:48:17 * SpamapS forgot about that 19:48:26 looks like it was working at least as of a few weeks ago 19:48:26 so, we can add ubuntu-xenial if needed 19:48:30 pabelanger: iirc that shouldn't matter because you set the chroot to xenial and its fine? 19:48:30 #link http://tarballs.openstack.org/packaging-deb/deb-nova/uploads/2081d05921eafc724ccb99a7b02c2e58f42ddf26/ 19:48:34 then create a repo in reprepro 19:48:35 should be relatively straight forward. 19:48:40 pabelanger: which is what we tried to get zigo to do in reverse when setting things up? 19:48:48 clarkb: right, we'd need to write the build scripts for that 19:49:16 oh i see the builds themselves are hardcoded to be debian-jessie 19:49:21 sorry - the sprinkler guy arrived right as this topic started 19:49:22 they also run on debian-jessie instances 19:49:28 right 19:49:41 so, it would take a little work to move this to ubuntu, but I can help do that 19:49:48 * mordred can help pabelanger 19:49:53 the running on debian-jessie is fine, we just need to make the builds target xenial 19:50:02 yes 19:50:13 i wouldn't be surprised if there's a fair amount of the pkg-deb team's workflow assumptions baked into those jobs right now (as far as the repo being a fork of what's being packaged, specific branch naming, et cetera) 19:50:28 likely need to make some changes to our JJB publishers, so we don't upload to openstack debian repos 19:50:41 that too 19:51:55 right now we just need bubblewrap packaged for whatever distro/release we're going to be running zuul executors on, right? 19:52:45 centos / fedora are okay today 19:53:10 and vhd-util for wherever we want to run nodepool builders? 19:53:15 fungi: yup 19:53:26 and vhd-util doesn't have any distros that work since its a super local patch right? 19:53:30 yes 19:53:46 there is no work that I'm aware of for getting that patched vhd-util into a distro in a reasonable manner 19:53:56 i gather it's not anything any sane distro would ever carry directly due to being a fork of xen or something 19:54:03 I've also heard rumors that modern qemu-img can make working vhd images 19:54:15 but some rummors of rackspace using qcow2? or qemu-img? 19:54:17 but as of yet that has not been verified by anyone here 19:54:32 pabelanger: no- just that qemu-img can convert raw to vhd that work on rax 19:54:41 ack 19:55:09 fungi: and then yes, getting bubblewrap for xenial is, I think, the other thing 19:55:20 also, on the bubblewrap front, debian does have packages (and backported to jessie too). we wouldn't necessary have to _run_ debian to use those, we could consider just using those packages on trusty or xenial since their dependencies can be met there with no trouble 19:55:21 so sounds like we need something regardless for vhd util 19:55:29 then we can decide if we want to piggy back bubblewrap on that 19:55:41 or use $otherpackagesource/distro/whatever 19:55:50 well - we can try to get qemu-img working for vhd and maybe drop our need for vhd-util 19:56:06 I'll do an upload today 19:56:12 but yes, if that still doens't work (and yuck long iteration cycle) - we need a place to put vhd-util 19:56:15 that might be a good first question to answer 19:56:23 curous if anyone has tried running the bubblewrap package from jessie/backports or from stretch 19:56:24 since if we need it for vhd-util, may as well do the same for bwrap. 19:56:32 but otherwise, lots more interesting options. 19:57:35 mordred: do you mind if we punt your other topic to next week? or to the ml (feedback from current dib contributors would be nice to get first anyway) 19:59:09 to sum this up, if newer qemu-img and cross-distro bubblewrap solve these issues, then i wouldn't worry about $topic for now. otherwise we probably want a spec 19:59:46 well, s/$topic/automating package builds and hosting our own/ 20:00:04 and that's all we have time for today 20:00:14 thanks everyone! 20:00:16 #endmeeting