19:01:15 #startmeeting infra 19:01:16 Meeting started Tue Jul 17 19:01:15 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:20 The meeting name has been set to 'infra' 19:01:31 p/ 19:01:34 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:44 #topic Announcements 19:01:56 I've been reminded to remind you all to submit to the summit CFP 19:02:00 I think you have just under 12 hours now 19:02:12 in any case it closes soon 19:02:51 yup 11:59 PST today which is 11 hours and 57 minutes from now 19:03:16 #topic Actions from last meeting 19:03:27 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-07-10-19.01.txt minutes from last meeting 19:03:56 ianw and fungi took an action to begin a letsencrypt POC. I don't expect this is done yet particularly with people traveling and on PTO 19:04:17 yeah, i'm more focused on the mm3 poc first 19:04:17 ianw: fungi: anything worth discussin on this item yet? 19:04:31 not yet, but yeah, i was on pto all last week 19:04:35 and about to disappear for ~two weeks starting tomorrow 19:04:53 but i can pitch in on whatever ianw starts when i get back, if he gets time 19:05:25 #action ianw fungi Set up letsencrypt POC 19:05:34 #topic Specs approval 19:05:43 #link https://review.openstack.org/565550 Containerized infra 19:05:56 two weeks ago I asked that we do our best to review this spec so that it can be put up for approval this week 19:06:27 A number of us have reviewed it and there are a few +1s. Any objections to formally putting it up for approval? I'd likely get around to doing that thursday afternoon local time 19:07:08 no objection from me 19:07:21 i reread that today. lgtm 19:08:37 i think it's ready 19:08:40 alright not hearing any objections. I will look forward to your final review prior to hopefully approving it on thursday 19:08:55 woot 19:09:33 also, relatedly, we're very close to having jobs building containers of zuul and nodepool - so some progress on parts of it have been made 19:09:54 cmurphy has also been pushing on the getting us out of the puppet hole tasks 19:10:13 ya 19:10:50 continuing to review topic:puppet-4 would be helpful (I try to dig into that periodically myself) 19:10:50 yay puppet hole 19:11:03 yes please 19:11:04 I'm expecting we are close to being able to pick a service for using puppet 4 19:11:34 yes, i proposed https://review.openstack.org/582765 to try to turn on the future parser for review-dev.o.o 19:11:55 #link https://review.openstack.org/582765 turn on future parser (puppet 4 parser) for review-dev.o.o 19:11:59 most of the other patches are just adding tests or are xenial fixes 19:12:30 there are some scary iptables changes that should probably be babysat 19:13:16 I found that the drush puppet module we're using was never updated for xenial + php7 so groups.o.o might be tough to update 19:13:31 cmurphy: we can probably loop in smarcet to help with that 19:13:49 the the refstack module is kind of broken, i'm hoping the refstack people can help update https://review.openstack.org/#/c/557539/ 19:13:51 or mrmartin if he's interested 19:14:07 (on the drush/groups front) 19:14:33 but mostly i think we should start taking the plunge and turning on the future parser as much as possible 19:15:00 as far as spec implementation planning goes, do we think there are a specific set of tasks that we want done before the ptg so that we can focus on other tasks face to face? 19:15:12 mordred: ^ you may have this paged in better than anyone 19:15:19 cmurphy: ++ 19:15:27 i think we can get all the puppet 4 stuff done before the ptg 19:16:24 clarkb: I think we should at least have split the run_all into individual playbooks - and it would probably be good to have an ansible version of openstack_project::server 19:16:48 but honestly, once we have openstack_project::server the rest of things are pretty parallelizable 19:17:42 clarkb: if I can ever get the pypi mirror working again, I'll start in on the base ansible things 19:17:46 I’m in for xenial / php7. 19:17:48 to summarize then we can get the puppet 4 work done so that we can limp along while we then parallelize ansible container conversions after having a version of openstack_project::server in ansible 19:18:17 order is roughly puppet 4 and openstack_project::server in ansible in parallel, then we can convert services to containers after ansible is done 19:18:52 ++ 19:18:55 I agree with those words 19:19:07 I want to say we are in good shape for the puppet stuff as long as we can get reviews (thank you cmurphy). Does anyone have the base server playbook on their radar? 19:19:37 oh, that's what I was going to work on after this container job 19:19:38 clarkb: maybe that warrants an #info ? :) 19:19:44 corvus: ++ 19:20:22 i'd like to pitch in a little on the base server, but i'm not prepared to bootstrap it 19:20:33 #info Puppet 4 work and openstack_project::server conversion to ansible can happen in parallel and are our prereqs. Once the ansible base playbook work is done we can parallelize per service container conversions 19:20:53 I can get it started, then will ping corvus when there's something there to rage-code against 19:21:02 #action mordred to begin work on openstack_project::server conversion to ansible with corvus helping 19:21:04 cool 19:21:31 #info review topic:puppet-4 to help get the puppet work completed 19:22:27 We sort of blended together the config mgmt priority effort and the spec approval topic. If there isn't anything else I'll jump to storyboard next 19:23:29 #topic Priority Efforts 19:23:38 #topic Storyboard 19:24:04 I've seen patches from Fatema go by in #storyboard. If you are interested in javascripty things reviews on those are likely helpful 19:24:21 We are also seeing interest from starlingx users 19:24:30 and airship too 19:24:45 but yesterday was starlingx questions, yes 19:25:42 there's an ML thread about openstack/requirements and needing to track tasks across storyboard and launchpad projects 19:25:48 Will be interesting to see what sort of feedback they come back with as very new users without storyboard history background 19:25:49 I almost responded to it- but didn't 19:26:45 More a heads up that there is stuff happening there than anything else. Fatema's changes in particular are aimed at improving search in storyboard which si something our users have wanted 19:26:52 yeah, i keep meaning to follow up on that one because the vmt has some similar patterns 19:27:04 the requirements sb thread i mean 19:27:18 clarkb: ++ 19:27:37 the obvious answer is what the vmt went with (live with having two different places you're copying stuff when it affects projects across different trackers) 19:28:05 fungi: yah - my answer was shaping into "we didn't really intend for the 'integrated release' projects to span both places for a long time" - but I realize that wasn't super helpful advice 19:28:13 I think fungi's VMT answer is likely the better answer 19:28:22 it's not a great solution, but it's less work than trying to engineer a complex solution for what should hopefully be a transitional situation 19:28:29 since it's at least based on what someone is actually doing 19:28:45 we didn't originally plan for integrated release projects to span both places 19:28:46 lp does know how to track external bugs, possible we could hack on top of that? 19:28:50 at least until we stop using lp 19:29:24 but the situation we ended up with is that different integrated release projects have different development workflows, use their defect and task tracking solutions in different ways, and are ready to transition at different times 19:29:44 so telling some of them that they can't have storyboard until all of them can have storyboard was a worse situation 19:30:49 ya we compromised in the opposite direction of making this particular problem easy 19:30:59 yes, lp has a feature where you can tell it your bug tracker is at some other url. i wasn't able to get that to actually work but the lp devs assured me in the last forum session we had that it does something so should find time to try again 19:31:23 fungi: I've used it when submitting python package bugs against ubuntu and for the python bug tracker it seemed to wrok 19:32:05 Any other storyboard related items before moving on? 19:32:16 yeah, i think when i tried a year-ish ago it was built around the idea that they needed to program a driver to figure out how to find bug reports in your particular tracker and they'd implemented it for some popular ones like bugzilla and debbugs 19:32:39 but maybe that's changed 19:33:00 that is the last state of it that I was aware of 19:34:38 #topic General Topics 19:35:07 Really quickly before we get to corvus' topic, we haven't forgotten about winterscale. I'm told jonathan will be attempting a followup today 19:35:46 I've also got a service breakdown proposal going at https://etherpad.openstack.org/p/winterscale-service-branding 19:35:58 your input on that is much apprecitated 19:36:03 (also I can't type) 19:36:07 * Shrews proposes "winterscale is coming" as a motto 19:36:18 ++ 19:36:39 alright corvus: swift and logs 19:37:34 #link http://lists.openstack.org/pipermail/openstack-infra/2018-July/006020.html thread on moving zuul job logs into swift 19:38:36 there's been some good discussion in email; mostly wanted to see if folks had feedback or questions in irc 19:38:36 * mordred welcomes our new logs-in-swift overlords 19:38:39 I'll try to summarize. Zuulv3 makes it a bit easier for us to talk to swift from zuul through the use of encrypted secrets. We've also found the ara middleware can crash our logs server in some cases due to use of memory leading to swapping 19:38:56 This leads to rebooting which potentially will require many hour long fscks 19:39:25 well, on friday, ext4 did its thing well and no fsck was needed 19:40:06 but it's a possibility, and we've still got potential iscsi failures looming over us (which always require a fsck) 19:40:15 i never want to do another of those 19:40:32 as for feedback I think we should probably identify which of our clouds provide an acceptable swift deployment and work with one of them to start. Then we can test things like how slow is uploading 30k small files for example 19:40:46 clarkb: is 30k typical or max? 19:41:10 corvus: I think that is more like the max, but we can grab a few ara db files and generate the files to check 19:41:20 to get an idea of what it is like for our current jobs 19:41:31 good idea. maybe for some typical devstack jobs or something. 19:41:36 ++ 19:41:45 i support the move to swift idea, in the past I did something similar jenkins logs to S3 instead of jenkins, to offload the master. obviously that this also depends on how scalable is our swift, maybe experiment with parallel usage first. 19:41:51 and multinode as that may potentially create a file count multiplier 19:41:59 seems like the massive ones were tripleo and openstack-ansible which were including their own ara content? 19:42:08 fungi: ya 19:42:23 did they change to using middleware too? 19:42:30 I believe tripleo did 19:42:37 not sure about osa 19:42:43 k. we should test that in particular then. 19:43:12 tbh, i'd really like us to favor the idea that we should get out of the middleware game. 19:44:19 corvus++ 19:44:24 ++ 19:45:01 i suppose an ara javascript front-end which retrieves an sqlite file and operates locally on it is a no-go ;) 19:45:03 I'm in favor as long as it doesn't create a big impact particularly for already very short jobs 19:45:16 regarding ara, i think it may be changed to lower its footprint in both total size and number of files. i was planning to investigate that but didn't hat time. 19:45:30 I'm less concerned if 3 hour tripleo job needs 10 minutes to upload files than 3 minute pep8 job needing 10 minutes to upload files 19:46:31 fungi: i'm not aware of js parsing sqlite libraries. but if there were a way to store data in json or something js can deal with, maybe that'd be a possibility. or maybe ssbarnea1's idea. :) 19:46:36 at leas in theory javascript can work directly with sqlite files, meaning that instead of tons of json files on disk, ara could just download the sqlite file and get data from it. 19:46:58 ssbarnea1: oh that's awesome! that may be something worth looking into 19:47:21 fungi, corvus: https://github.com/kripken/sql.js/ 19:47:27 i don't know what the ara middleware does right now; but i imagine a solution like that would look like implementing whatever that is in js... 19:47:45 anyway, bit of a rabbit hole 19:47:48 sorry 19:48:05 it's worth talking about. it's definitely the weak point in the plan. 19:48:24 ++ 19:48:29 my argument is basically we should accept the cost as long as it's reasonable, because it's a net benefit. but if we can minimize the cost, or even reverse it, great! 19:49:04 fwiw similar discussion has come up re kata using zuul 19:49:19 they wanted to know how to make certain things htmly and the suggestion mnaser and I had was to write html that gets uploaded 19:49:44 I also think outside of general zuul reporting the people best prepared to present their logs are the ones producing them 19:50:11 i think that's a good suggestion. it gives folks local control and the system overall flexibility. 19:50:14 what we've done with os loganalyze is sort of a "here is a best effort thing that adds some color based on a guess" 19:51:25 in my message to the zuul list, i do, however, suggest that we can js-ify osla and make it generic. i think that might be a nice thing to do that would help out people in general, and probably handle a lot of the typical use cases. 19:51:48 yah. html-ifying log files with timestamps seems doable and friendly 19:52:01 corvus: I do like that for very consistent and specific outputs like journald logs 19:52:07 ++ 19:52:21 with python and openstack in particular we have never had that kind of consistency though 19:53:34 Running out of time. Lets talk project renames really quickly 19:53:39 #topic Project Renames 19:54:34 I sort of punted on this a couple weeks ago to see where we would be at today. I think at least one of us is on vacation over the next couple weeks. Rocky release schedule makes it look like July 30- August 3 is our best bet 19:55:00 I expect I will be around taht week (parents are flying in night of the 2nd though) 19:55:39 Anyone else expect to be around on say August 3 at 1600UTC? 19:55:58 if so I can run it by the release team and see if that works 19:56:10 i'll be around 19:56:17 count me in 19:57:12 #info Project renames tentatively scheduled for August 3 at 1600UTC. 19:57:18 #topic Open Discussion 19:58:44 do we want to set up a time to babysit https://review.openstack.org/582765 and maybe a few others and start making headway on that? 19:58:50 i have one subject, reviving few dormant projects 19:59:18 cmurphy: I can help with that on Friday probably 19:59:24 (I try to keep friday's clear for that sort of thing) 19:59:26 as mentioned earlier, i have a mailman3 proof of concept going on. notes at: 19:59:27 ssbarnea1: go for it 19:59:28 #link https://etherpad.openstack.org/p/mm3poc 19:59:39 clarkb: friday's not good for me :/ 19:59:46 if anybody's interested in helping with it, i've got it most of the way there 19:59:49 cmurphy: would thursday work instead? I can be flexible 19:59:53 thursday works 19:59:53 but am disappearing for the next two weeks 19:59:54 examples: gerritlib, git-review, bashate. I would like to help with them and I thin that electrofelix also wants the same. 19:59:56 but today and tomorrow probably need time to review more 20:00:15 all infra-root ssh keys should be accepted by lists-dev01.o.o 20:00:27 at least one of them has a review that is 36 months old! 20:00:34 ssbarnea1: yes there was the mailing list thread for git-review. I think that helped set expectations at least for that project? 20:01:39 indeed, i need to recheck the thread as I do not remember what was the conclusion. 20:01:52 bashate isn't an infra project iirc it is a qa project. Gerritlib is definitely a case of it does what we need ti to do and otherwise doesn't get much attention but probably should. There is also an alternative implementation on pypi that maybe we should consider moving too? Unsure if that would work for zuul 20:02:10 corvus: ^ zuul may have interest in gerritlib in particular as a dep? 20:02:11 ssbarnea1: yes, for bashate, i merged a few things and will do a release 20:03:09 Not to kill the conversation, but I do have a meeting in under half an hour I need to prep for. I'll stop the meeting here, but feel free to continue discussion in #openstack-infra or on the mailing list 20:03:13 thank you everyone 20:03:15 #endmeeting