| *** annabelleB has joined #openstack-tc | 00:35 | |
| *** annabelleB has quit IRC | 00:37 | |
| *** diablo_rojo has quit IRC | 00:39 | |
| *** annabelleB has joined #openstack-tc | 02:22 | |
| *** melwitt has joined #openstack-tc | 02:24 | |
| *** annabelleB has quit IRC | 02:31 | |
| *** lbragstad_503 has quit IRC | 05:27 | |
| *** lbragstad_503 has joined #openstack-tc | 05:27 | |
| *** annabelleB has joined #openstack-tc | 05:48 | |
| *** annabelleB has quit IRC | 06:08 | |
| *** openstackgerrit has joined #openstack-tc | 06:35 | |
| openstackgerrit | Tony Breeds proposed openstack/governance master: T Release Name https://review.openstack.org/611511 | 06:35 |
|---|---|---|
| *** e0ne has joined #openstack-tc | 06:38 | |
| *** ricolin has joined #openstack-tc | 06:40 | |
| *** persia has quit IRC | 07:19 | |
| *** persia has joined #openstack-tc | 07:21 | |
| evrardjp | o/ | 07:28 |
| *** tosky has joined #openstack-tc | 07:31 | |
| *** openstackgerrit has quit IRC | 07:35 | |
| *** dtantsur|afk is now known as dtantsur | 07:56 | |
| *** jpich has joined #openstack-tc | 08:00 | |
| *** e0ne has quit IRC | 08:02 | |
| *** e0ne has joined #openstack-tc | 08:13 | |
| *** ricolin has quit IRC | 10:29 | |
| *** ricolin has joined #openstack-tc | 10:59 | |
| *** e0ne has quit IRC | 11:50 | |
| *** dims has quit IRC | 12:30 | |
| *** dims has joined #openstack-tc | 12:33 | |
| *** mriedem has joined #openstack-tc | 12:38 | |
| *** e0ne has joined #openstack-tc | 13:07 | |
| *** cdent has joined #openstack-tc | 13:08 | |
| *** lbragstad_503 is now known as lbragstad | 13:32 | |
| *** openstackgerrit has joined #openstack-tc | 13:51 | |
| openstackgerrit | Corey Bryant proposed openstack/governance master: Add optional python3.7 unit test enablement to python3-first https://review.openstack.org/610708 | 13:51 |
| mriedem | ttx: do you know if there are still unused forum session slots available? | 14:30 |
| ttx | I was not on the selection committee... I think I heard they allocated all their slots, let me see | 14:30 |
| mugsie | mriedem: all slots were allocated | 14:31 |
| mriedem | ok | 14:31 |
| ttx | we could potentially change things / merge your topic in a session though, I guess | 14:31 |
| mriedem | i know lyarwood had duplicate sessions, | 14:32 |
| mriedem | but yeah | 14:32 |
| *** annabelleB has joined #openstack-tc | 14:34 | |
| *** annabelleB has quit IRC | 14:40 | |
| *** cdent has quit IRC | 14:54 | |
| fungi | openstack community: tc-members are around to answer your questions for the next hour | 15:00 |
| lbragstad | o/ | 15:00 |
| gmann | o/ | 15:00 |
| gmann | sent email too | 15:04 |
| gmann | btw i wanted to discuss the gate job migration to bionic | 15:04 |
| gmann | frickler and myself was discussing about migration of jobs to Bionic on qa channel today | 15:05 |
| *** dklyle has joined #openstack-tc | 15:06 | |
| dims | o/ | 15:07 |
| gmann | As most of the base job QA team own(in devstack and tempest), I agree (from QA PTL point of view) on point that QA team needs to take the ownership of migrating those jobs to new platform. | 15:07 |
| gmann | we have started the progress in that. | 15:07 |
| gmann | #link https://etherpad.openstack.org/p/devstack-bionic | 15:07 |
| fungi | thanks gmann! | 15:08 |
| *** jeremyfreudberg has joined #openstack-tc | 15:08 | |
| gmann | Plan is 1. test the projects side and integration jobs to Bionic (^^ etherpad started by frickler) 2. Fix the issues if any 3. if all well, QA team will migrate the base job setup | 15:09 |
| gmann | and also work with projects specific job testing before we migrate the base jobs setup | 15:09 |
| gmann | is this plan works fine ? any feedback/opinion | 15:10 |
| gmann | and another question is - is it ok to do this migration in stein ? | 15:10 |
| gmann | from TC perspective | 15:10 |
| *** annabelleB has joined #openstack-tc | 15:10 | |
| zaneb | so I would say I'm +1 personally on doing this in Stein, but I think we should have that recorded somewhere and the whole TC vote on it (https://review.openstack.org/611080 seems like a good place to me) | 15:12 |
| *** e0ne has quit IRC | 15:12 | |
| mugsie | we do say "latest LTS" on https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions | 15:12 |
| tosky | gmann: and tell users of the devstack-tempest job to keep an eye on their jobs | 15:13 |
| gmann | tosky: yeah, we are going to float the ML on that | 15:13 |
| gmann | zaneb: mugsie yeah it is already there as "latest LTS". do we need anything else to make it very specific and explicitly ? | 15:14 |
| mugsie | I don't remember anything like that for the xenial migration, it just sort of happened akaik | 15:14 |
| gmann | tosky: we are not going to merge the base job changes until all projects side jobs are tested well. | 15:14 |
| mugsie | afaik* | 15:14 |
| fungi | in the past the tc hasn't really voted on it, we've just been able to rely on the openstack infra team to push the community to upgrade testing. glad to see the qa team taking the lead this time (it certainly seems within their remit, even more than it was for the infra team) | 15:14 |
| gmann | yeah | 15:14 |
| zaneb | mugsie: yeah, that's fair. so I guess there is no obstacle to the QA team going ahead | 15:15 |
| fungi | mugsie: yeah, we've tried a couple solutions in the past. for precise to trusty the infra team just put on their flame-retardant suits, flipped the switch, let everything break, and let the teams work out the blockers in their particular jobs | 15:16 |
| mugsie | fungi: I remember :) | 15:16 |
| zaneb | that said, the QA team are not the only folks running functional test jobs, and if we want to inspire concerted action from everyone, just having the general principle documented will not do it | 15:16 |
| fungi | for trusty to xenial, because everyone complained about the breakage the previous time, we told teams to migrate at their own pace piecemeal. that also didn't work out well because a number of teams procrastinated | 15:16 |
| fungi | and so we released openstack with integration testing split between two different ubuntu lts versions, and stuff broke that way instead | 15:17 |
| fungi | i should point out that the precise->trusty switch was announced well ahead, experimental jobs were set up and teams were encouraged to work out their issues ahead of the flag day | 15:18 |
| *** e0ne has joined #openstack-tc | 15:18 | |
| fungi | really most of the pain from either method came down to some teams procrastinating and prioritizing other work. so not terribly surprising | 15:19 |
| mugsie | yeah, either way there will be pain on some side | 15:19 |
| gmann | this time (with zuulv3 jobs) base jobs are the one defining the ubuntu version (in 90% of the job if i remember ). so changing there is going to effect most of the projects side jobs too | 15:19 |
| zaneb | fungi: iirc both of those occurred before we had a project-wide goals process. do you think following that process (with e.g. goal champions &c.) might make it go smoother? | 15:20 |
| gmann | but yes, any failure which is project specific job side has to be fixed by project team (QA and infra team can alwys help) | 15:20 |
| fungi | zaneb: i think following the goals process for something like this does at least provide us with a bit better insight into the state of progression for the switch across the projects | 15:20 |
| mugsie | gmann: is there a devstack-tempest-bionic we can tell teams to use to check things, and then we can swap the OS of the root job after $TIME ? | 15:20 |
| gmann | mugsie: yes, it is there - https://review.openstack.org/#/c/610977/ | 15:21 |
| gmann | and this testing all the tempest gate jobs -https://review.openstack.org/#/c/611572/ | 15:21 |
| fungi | zaneb: which is why i'm in favor of smcginnis_vaca's governance change (in some form) being positioned as a fait accompli cycle goal | 15:22 |
| gmann | and etherpad tracking the project specific jobs testing - https://etherpad.openstack.org/p/devstack-bionic | 15:22 |
| zaneb | fungi: yes, I am also +1 on that but I think it needs to say how to achieve the goal | 15:23 |
| *** cdent has joined #openstack-tc | 15:23 | |
| zaneb | the current patch only records what the latest LTS version was at the start of the cycle | 15:24 |
| *** jamesmcarthur has joined #openstack-tc | 15:24 | |
| fungi | i guess "jdi" isn't enough to say how? ;) | 15:24 |
| clarkb | zaneb: I think start of cycle is the critiera? | 15:25 |
| clarkb | zaneb: bionic released during the rocky cycle but because we were halfway through it deferred to stein | 15:25 |
| clarkb | s/it/rocky/ | 15:25 |
| fungi | we never explicitly said anywhere formal that it was latest lts "as of the start of that cycle" but we've so far interpreted it that way to minimize disruption to the release process | 15:25 |
| clarkb | ah | 15:26 |
| fungi | the pti doesn't get quite that specific | 15:26 |
| cdent | I think we should state it as lts at the _end_ of the cylce | 15:26 |
| cdent | (expected lts) | 15:26 |
| *** annabelleB has quit IRC | 15:27 | |
| fungi | but we've also got the counterpoint from distro packagers that we should be targeting what the distros plan to release rather than targeting what already exists | 15:27 |
| zaneb | clarkb: my point was it's useful to say "to save you looking it up, the latest Ubuntu LTS is 18.04", but more useful to say "you need to update these specific test jobs before the end of Stein" | 15:27 |
| mugsie | yeah, that is more difucult though | 15:27 |
| fungi | cdent: running jobs on bionic well before bionic existed would have been challenging | 15:27 |
| mugsie | we have no idea what is in RHEL $VERSION+1, or when exactly debian will be released | 15:28 |
| clarkb | fungi: though possible because of the beta releases which pabelanger had images running for | 15:28 |
| fungi | cdent: as i mentioned in the review, the solution to that is to run on the rolling release that ubuntu freezes from, so basically debian/testing or debian/unstable | 15:28 |
| cdent | fungi: I think you're just being pedantic, but what I mean is: the python that is expected to be in the LTS at the end of the release, even if that means running a non-lts | 15:28 |
| clarkb | more difficult with rhel/centos because well they just drop | 15:28 |
| cdent | fungi, yes, that | 15:28 |
| fungi | clarkb: we didn't have bionic beta releases at the start of the rocky cycle | 15:28 |
| gmann | for ownership. i will say - each repo defining those jobs has ownership. QA for most of them as they are providing the base jobs setup | 15:29 |
| fungi | clarkb: and yeah, we can't really test open0core distros that lack an open release process | 15:29 |
| fungi | er, can't test them in advance of releasing that is | 15:29 |
| fungi | we could test on fedora rawhide which is sort of that for rhel i suppose | 15:29 |
| zaneb | cdent: you're mixing up unit and functional tests there. you can't always treat them the same | 15:29 |
| cdent | zaneb: where am I mixing that up? | 15:30 |
| *** devananda has joined #openstack-tc | 15:30 | |
| fungi | cdent: i'm not being pedantic, i'm saying not everything we produce is in python, and not bug we're going to find will be related to the python interpreter (many if not most will be things like newer qemu/kernel, c libs linked in our deps, system tools we call into from subprocesses, et cetera) | 15:31 |
| zaneb | cdent: it's generally easy to do unit tests with the version of python you expect in the next LTS. It's much harder to do functional tests on a distro that hasn't been released yet. The discussion started with gmann talking about functional testing | 15:31 |
| cdent | zaneb: you asked for a general principle of what I'm going for, which I haven't had a chance to respond to on the review, but the gist is this: upstream should work slightly in the future of distributors and one way to signal this is by testing with an OS that provides the Python that is the default distributed from Python from their front page (currently 3.7.0). | 15:33 |
| *** annabelleB has joined #openstack-tc | 15:33 | |
| cdent | I acknowledge that this will break things | 15:33 |
| cdent | breaking things is what tests are for | 15:33 |
| cdent | so that we can fix things | 15:33 |
| clarkb | as far as method goes ya I think devstack/tempest should start by adding the bionic nodesets and then push a self testing change that bumps nodeset on base devstack/tempest jobs to bionic | 15:33 |
| cdent | we've gotten ourselves into a state where we want the tests to always pass and to not have to fix them unless we change our code. we need to use testing (and CI in general) as a forcing function for focussing our work into the future | 15:34 |
| clarkb | that change will be self testing and once it passes set a day to merge it. Chances are corner cases will break in projects consuming that and we'll work through it from there (hence setting a day to merge it) | 15:34 |
| gmann | clarkb: yeah, we started that - https://etherpad.openstack.org/p/devstack-bionic | 15:34 |
| cdent | which is where we (upstream) live | 15:34 |
| clarkb | gmann: great! | 15:34 |
| zaneb | cdent: there wasn't a platform on which we could have tested that at the beginning of Stein, and not because OpenStack was broken | 15:34 |
| cdent | sure there was: debian had 3.7 available before rocky was released | 15:34 |
| zaneb | cdent: actually, I tell a lie. we could have tested it on Debian unstable | 15:34 |
| zaneb | yep | 15:34 |
| clarkb | cdent: zaneb for python3.7? | 15:34 |
| clarkb | fedora 28 has it today and we have images for that too aiui | 15:35 |
| clarkb | and now bionic also has packages for it too | 15:35 |
| cdent | anyway: as I said in one of my screeds: I'm not wed to this idea, it may be impractical. But I _am_ wed to the principles it implies: that we should be testing for what it is to come, not what has already passed | 15:35 |
| zaneb | clarkb: F28 version was a beta that crashed similar to U18.04 version, at least at the beginning of Stein | 15:36 |
| clarkb | ++ this is one reason I've encouraged dirk and others to get the tumbleweed image up and running | 15:36 |
| clarkb | it is an actual rolling release distro with CI itself (unlike arch) | 15:36 |
| clarkb | where we can in theory test new versions of all the things on a rolling basis | 15:36 |
| clarkb | though its still on python3.6 for some reason | 15:37 |
| cdent | zaneb: when you said "It's much harder to do functional tests on a distro that hasn't been released yet. " did you mean integration, not functional? | 15:39 |
| cdent | (that is, tempest, not in-tree functional tests) | 15:39 |
| zaneb | cdent: I... don't know because everyone uses different terminology | 15:39 |
| fungi | we don't even seem to always agree on what count as "unit tests" | 15:40 |
| cdent | true enough | 15:40 |
| zaneb | Heat's 'functional' tests are in-tree but tempest-like, so... | 15:40 |
| * zaneb shrugs | 15:40 | |
| fungi | also, i'm stepping afk for a bit. replacement appliances just arrived | 15:40 |
| mugsie | yeah, we dropped our functional tests and went full tempest | 15:40 |
| cdent | when I say "integration" I mean "live processes started outside the running of the tests" | 15:40 |
| zaneb | cdent: yeah, I meant that | 15:41 |
| cdent | when I say "functional" I mostly mean "yeah, there's a database there, but it is probably in memory" | 15:41 |
| cdent | (which I agree is much too vague) | 15:41 |
| * zaneb should just stop saying functional | 15:43 | |
| clarkb | I definitely think if there is interest it would be a worthwhile experiement to run a visible job (maybe this means voting or maybe non voting in a lot of places?) on top of a "prerelease" type distro. Either add debian unstable as an option or use tumbleweed which is already there (or $other thing, maybe gentoo which is mostly there now too) | 15:43 |
| clarkb | I expect this would be very valuable for nova and neutron in particular because they both interact with the system in interesting ways | 15:44 |
| *** jeremyfreudberg has quit IRC | 15:44 | |
| gmann | ok, i think we all are ok for Bionic migration plan for stein. and we can work to make "such work/migration" a process for future (when and who) in this- https://review.openstack.org/#/c/611080/ | 15:50 |
| cdent | clarkb, fungi: What are your thoughts (from an infra standpoint) on the need to limit the number of new jobs, or have some kind of "add one, remove one" policy? | 15:52 |
| clarkb | cdent: typically that type of stuff doesn't concern me too much because the vast majority of resource usage is from such a small set of projects that most projects can add and remove jobs as they like without making a major impact | 15:53 |
| cdent | i'm thinking more in terms of removing templated jobs | 15:54 |
| clarkb | if we are actually concerned about resource usage then combatting the incredibly flaky gate problem and disproportionate use by a small number is the way to do that | 15:54 |
| cdent | ones that many projects use | 15:54 |
| clarkb | cdent: still not a huge concern, tripleo neutron and nova are most of our resource usage, like 80% ish | 15:54 |
| gmann | i would like to go for second approach | 15:55 |
| cdent | I definitely agree that we need to work harder with regards to the flakey gate problem | 15:55 |
| clarkb | we are also in a (hopefully) temporary degraded state with resource allocations from ovh as they work through their post upgrade stuff | 15:58 |
| clarkb | and the packethost/platform9 donation hasn't quite stabilized yet | 15:58 |
| clarkb | I'm sure some of that is related to the software we produce so finding better ways to communicate that across might help too? | 15:59 |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| cdent | I sometimes wish there was a way to break the gate completely for a project once it crosses a threshold of N units of flakiness. Whatever that is. We are inured to flakiness more than is good for us. | 16:00 |
| cdent | Anyway, I gotta go. | 16:00 |
| * cdent waves | 16:00 | |
| *** cdent has quit IRC | 16:00 | |
| *** e0ne has quit IRC | 16:00 | |
| * lbragstad steps away to get a run in over lunch | 16:01 | |
| *** mriedem is now known as mriedem_lunch | 16:10 | |
| *** annabelleB has quit IRC | 16:12 | |
| *** jamesmcarthur has quit IRC | 16:15 | |
| *** dklyle has quit IRC | 16:16 | |
| *** dklyle has joined #openstack-tc | 16:16 | |
| *** cdent has joined #openstack-tc | 16:24 | |
| *** ricolin has quit IRC | 16:25 | |
| *** jamesmcarthur has joined #openstack-tc | 16:30 | |
| *** jpich has quit IRC | 16:30 | |
| *** dklyle has quit IRC | 16:37 | |
| *** annabelleB has joined #openstack-tc | 16:39 | |
| *** annabelleB has quit IRC | 16:43 | |
| *** annabelleB has joined #openstack-tc | 16:52 | |
| *** dklyle has joined #openstack-tc | 17:30 | |
| *** dklyle has quit IRC | 17:42 | |
| *** diablo_rojo has joined #openstack-tc | 17:44 | |
| *** dklyle has joined #openstack-tc | 17:50 | |
| *** mriedem_lunch is now known as mriedem | 17:51 | |
| *** dklyle has quit IRC | 18:02 | |
| *** annabelleB has quit IRC | 18:07 | |
| *** jamesmcarthur has quit IRC | 18:09 | |
| *** diablo_rojo has quit IRC | 18:12 | |
| *** diablo_rojo has joined #openstack-tc | 18:13 | |
| *** lbragstad has quit IRC | 18:17 | |
| *** dklyle has joined #openstack-tc | 18:20 | |
| *** annabelleB has joined #openstack-tc | 18:25 | |
| *** diablo_rojo has quit IRC | 18:26 | |
| *** scas has quit IRC | 18:28 | |
| *** dklyle has quit IRC | 18:29 | |
| *** annabelleB has quit IRC | 18:32 | |
| *** annabelleB has joined #openstack-tc | 18:34 | |
| *** jamesmcarthur has joined #openstack-tc | 18:37 | |
| *** jamesmcarthur has quit IRC | 18:43 | |
| *** diablo_rojo has joined #openstack-tc | 18:44 | |
| *** lbragstad has joined #openstack-tc | 18:48 | |
| *** dklyle has joined #openstack-tc | 19:07 | |
| *** jamesmcarthur has joined #openstack-tc | 19:12 | |
| *** dklyle has quit IRC | 19:14 | |
| *** jamesmcarthur has quit IRC | 19:16 | |
| *** lbragstad has quit IRC | 19:21 | |
| *** lbragstad has joined #openstack-tc | 19:24 | |
| *** annabelleB has quit IRC | 19:32 | |
| *** diablo_rojo has quit IRC | 19:35 | |
| *** devananda has quit IRC | 19:47 | |
| *** e0ne has joined #openstack-tc | 19:52 | |
| *** e0ne has quit IRC | 20:05 | |
| *** zaneb has quit IRC | 20:31 | |
| *** openstackgerrit has quit IRC | 20:36 | |
| *** annabelleB has joined #openstack-tc | 20:54 | |
| *** zaneb has joined #openstack-tc | 20:54 | |
| *** zaneb has quit IRC | 21:11 | |
| *** diablo_rojo has joined #openstack-tc | 21:12 | |
| cdent | dims: how 'live' is https://review.openstack.org/#/c/586212/ | 21:32 |
| *** cdent has quit IRC | 21:34 | |
| *** dklyle has joined #openstack-tc | 21:36 | |
| *** diablo_rojo has quit IRC | 21:42 | |
| *** diablo_rojo has joined #openstack-tc | 21:44 | |
| *** dklyle has quit IRC | 21:45 | |
| *** openstackgerrit has joined #openstack-tc | 21:50 | |
| openstackgerrit | Merged openstack/governance master: Fix format errors in PTI docs https://review.openstack.org/611098 | 21:50 |
| openstackgerrit | Merged openstack/governance master: Update sphinx extension logging https://review.openstack.org/611132 | 21:50 |
| *** jamesmcarthur has joined #openstack-tc | 21:55 | |
| *** jamesmcarthur has quit IRC | 21:59 | |
| *** scas has joined #openstack-tc | 22:09 | |
| *** mriedem has quit IRC | 22:12 | |
| *** diablo_rojo has quit IRC | 22:52 | |
| *** tosky has quit IRC | 23:02 | |
| *** diablo_rojo has joined #openstack-tc | 23:03 | |
| diablo_rojo | Reminder that we have 3 days left to nominate people for Community Contributor Awards! It only takes a few minutes to nominate someone. https://openstackfoundation.formstack.com/forms/berlin_stein_ccas | 23:10 |
| *** zaneb has joined #openstack-tc | 23:24 | |
| *** annabelleB has quit IRC | 23:31 | |
| mnaser | It would have been nice to take this operating system discussion to the ML. | 23:33 |
| *** jamesmcarthur has joined #openstack-tc | 23:36 | |
| *** diablo_rojo has quit IRC | 23:37 | |
| *** jamesmcarthur has quit IRC | 23:39 | |
| *** jamesmcarthur has joined #openstack-tc | 23:39 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| *** jamesmcarthur has joined #openstack-tc | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!