| weshay | not yet | 00:06 |
|---|---|---|
| weshay | getting there | 00:06 |
| weshay | one package installed, but not the other | 00:06 |
| weshay | nice thing is.. I can get on the node and poke :) | 00:06 |
| weshay | rlandy, woot | 00:10 |
| weshay | ok.. wip/ugly reviews coming up | 00:10 |
| * rlandy looks for ugly reviews | 00:12 | |
| weshay | https://review.openstack.org/633628 | 00:13 |
| weshay | https://review.rdoproject.org/r/18586 | 00:14 |
| weshay | I'm in TASK [ansible-role-tripleo-ci-reproducer : prepare nodes] *************************************** | 00:15 |
| weshay | 00:15 | |
| weshay | atm | 00:15 |
| weshay | woot.. starting containers | 00:16 |
| weshay | still don't have logs but meh | 00:19 |
| rlandy | hmm .. trying your change | 00:20 |
| weshay | I have zuul containers failing atm | 00:21 |
| weshay | that happened w/ provider=cloud too | 00:21 |
| rlandy | running your changes now | 00:21 |
| weshay | maybe the containers will work on your system | 00:24 |
| rlandy | [ansible-role-tripleo-ci-reproducer : prepare nodes | 00:25 |
| rlandy | failure | 00:28 |
| rlandy | TASK [ansible-role-tripleo-ci-reproducer : prepare nodes] filed | 00:28 |
| rlandy | failed | 00:28 |
| weshay | oh.. but it got through all the setup | 00:32 |
| weshay | any more detail? | 00:32 |
| weshay | or we can look at it tomorrow | 00:32 |
| weshay | late for you | 00:32 |
| weshay | rlandy, I fired off a email to quiquell|off and ssbarnea|bkp2 | 00:32 |
| weshay | er.. sshnaidm | 00:32 |
| rlandy | spewed out this huge error | 00:33 |
| rlandy | reading through | 00:33 |
| rlandy | install clash | 00:33 |
| rlandy | but mom on phone ...talking ... talking | 00:33 |
| rlandy | ansible python module location = /usr/lib/python2.7/site-packages/ansible\n | 00:35 |
| rlandy | weshay: ^^ | 00:35 |
| rlandy | oh the mirror thing | 00:36 |
| rlandy | they were talking about it today | 00:36 |
| * rlandy runs again defining mirror | 00:36 | |
| rlandy | FAILED! => {\"changed\": false, \"msg\": \"\\n\\n One of the configured repositories failed (Unknown),\\n and yum doesn't have enough cached data to continue. At this point the only\\n safe thing yum can do is fail. There are a few ways to work \\\"fix\\\" this:\\n\\n 1. Contact the upstream for the repository and get them to fix the problem.\\n\\n 2. Reconfigure the baseurl/etc. | 00:37 |
| rlandy | mirror_fqdn | 00:38 |
| *** trown is now known as trown|outtypewww | 00:39 | |
| rlandy | -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 00:40 |
| rlandy | weshay: ^^ trying with that | 00:40 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message) | 00:47 |
| rlandy | ha ... TASK [ansible-role-tripleo-ci-reproducer : Start up zuul and friends] | 00:48 |
| rlandy | 1001: Add job to launch Depends-On: h Depends-On: t Depends-On: t Depends-On: p Depends-On: s Depends-On: : Depends-On: / Depends-On: / Depends-On: r Depends-On: e Depends-On: v Depends-On: i Depends-On: e Depends-On: w Depends-On: . Depends-On: o Depends-On: p Depends-On: e Depends-On: n Depends-On: s Depends-On: t Depends-On: a Depends-On: c Depends-On: k Depends-On: . Depends-On: o Depends-On: r Depends-On: g Depends-On: / | 00:51 |
| rlandy | Depends-On: 6 Depends-On: 3 Depends-On: 1 Depends-On: 0 Depends-On: 6 Depends-On: 7 — | 00:51 |
| rlandy | zuul.yaml | 00:51 |
| rlandy | ok - so that's not right | 00:52 |
| rlandy | but it's running | 00:52 |
| rlandy | fixing that | 00:57 |
| rlandy | https://review.openstack.org/631067 updated | 01:10 |
| rlandy | bbl | 01:14 |
| *** rlandy is now known as rlandy|bbl | 01:15 | |
| *** tosky has quit IRC | 01:34 | |
| *** fultonj has quit IRC | 02:07 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 02:47 |
| *** ykarel|away has joined #oooq | 02:55 | |
| *** rlandy|bbl is now known as rlandy | 03:04 | |
| *** rlandy has quit IRC | 03:04 | |
| *** gkadam has joined #oooq | 03:11 | |
| *** apetrich has quit IRC | 03:15 | |
| *** ykarel|away is now known as ykarel | 03:42 | |
| *** ykarel is now known as ykarel|afk | 03:50 | |
| *** udesale has joined #oooq | 04:02 | |
| *** ykarel|afk is now known as ykarel | 04:06 | |
| *** chkumar|out is now known as chandankumar | 04:22 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 04:47 |
| *** ratailor has joined #oooq | 05:23 | |
| *** udesale has quit IRC | 05:29 | |
| *** jtomasek has joined #oooq | 05:36 | |
| *** udesale has joined #oooq | 05:51 | |
| *** udesale has quit IRC | 05:59 | |
| *** udesale has joined #oooq | 06:00 | |
| *** jtomasek has quit IRC | 06:02 | |
| *** ykarel has quit IRC | 06:09 | |
| *** ykarel has joined #oooq | 06:21 | |
| *** ccamacho has quit IRC | 06:44 | |
| *** skramaja has joined #oooq | 06:45 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 06:47 |
| *** dsneddon has quit IRC | 06:59 | |
| *** quiquell|off is now known as quiquell | 07:01 | |
| *** dsneddon has joined #oooq | 07:04 | |
| *** udesale has quit IRC | 07:10 | |
| *** udesale has joined #oooq | 07:18 | |
| *** ccamacho has joined #oooq | 07:20 | |
| *** udesale has quit IRC | 07:22 | |
| *** udesale has joined #oooq | 07:26 | |
| quiquell | weshay, sshnaidm: Run centos7 repro ci at more clean image https://review.rdoproject.org/r/#/c/18593/ | 07:40 |
| quiquell | It's already breaks at pre.yaml so we are good | 07:40 |
| quiquell | Will put a review at software factory for f28 so we have clean fedora28 nodesets too | 07:41 |
| quiquell | Also python3 with fedors28 | 07:41 |
| *** jfrancoa has joined #oooq | 07:41 | |
| marios | jfrancoa: o/ hey man can you remove Gabriele Santomaggio https://review.openstack.org/#/c/633486/1 | 08:06 |
| marios | i addidentally spammed that poor person trying to add panda | 08:06 |
| chandankumar | sshnaidm: marios: quiquell panda https://review.openstack.org/#/c/633214/ https://review.openstack.org/#/c/633185/ | 08:06 |
| marios | jfrancoa: i can't cos its not my review | 08:07 |
| chandankumar | marios: regarding voting one I am updating a patchset | 08:07 |
| marios | chandankumar: ack will check in a bit | 08:07 |
| marios | chandankumar: ok - yeah do you want to ask in tripleo today or you happy for it to go voting & in gate? | 08:07 |
| marios | chandankumar: do we want it in the standalone template instead? | 08:07 |
| marios | chandankumar: i don't know man | 08:08 |
| chandankumar | marios: sure | 08:08 |
| marios | chandankumar: like does it needs more discussion first? | 08:08 |
| marios | before making it voting & in gate | 08:08 |
| chandankumar | marios: I will bring this up in tripleo meeting | 08:08 |
| marios | we can switch on the voting anyt ime | 08:08 |
| marios | is like 2 line review | 08:08 |
| chandankumar | marios: I am updating it with non-voting and also bring the discussion there | 08:08 |
| marios | chandankumar: ^ | 08:08 |
| marios | chandankumar: ack | 08:08 |
| *** apetrich has joined #oooq | 08:09 | |
| jfrancoa | marios: sure, no problem. Removed, I'm sure he was happy to have some new review and we spoiled his happiness :-D | 08:14 |
| marios | jfrancoa: :D | 08:16 |
| marios | (wow i have friends!) | 08:16 |
| marios | oh was mistake :0 | 08:16 |
| *** dsneddon has quit IRC | 08:19 | |
| *** kopecmartin|off is now known as kopecmartin | 08:19 | |
| quiquell | chandankumar: +w cirros | 08:22 |
| quiquell | chandankumar: maybe better to search cloud by name, you don't know what people is going to have a clouds.yaml | 08:22 |
| marios | quiquell: wow we are reviewing the same thing/same time :D | 08:24 |
| marios | quiquell: chandankumar patches | 08:24 |
| chandankumar | quiquell: marios ok, I will do that | 08:25 |
| quiquell | marios: we are entangled | 08:25 |
| marios | quiquell: reviewjobception | 08:25 |
| chandankumar | based on cloud name searching and making the job non-voting | 08:25 |
| marios | quiquell: chandankumar revoted and added commenthttps://review.openstack.org/#/c/633185/6 if you're lookup by cloud name then also restrict the cloud.yaml search | 08:28 |
| chandankumar | marios: quiquell thanks, updating it | 08:29 |
| *** jpena|off is now known as jpena | 08:33 | |
| quiquell | marios: looks like the proper mirror variable is zuul_site_mirror_fqdn | 08:41 |
| quiquell | marios: https://github.com/openstack-infra/zuul-jobs/blob/master/roles/configure-mirrors/defaults/main.yaml#L1 | 08:42 |
| quiquell | marios: re: dryrun | 08:42 |
| *** tosky has joined #oooq | 08:42 | |
| marios | quiquell: thank you i will update it in a bit | 08:44 |
| marios | quiquell: thanks man! | 08:44 |
| quiquell | marios: let's see what's the next issue with dryrun | 08:44 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 08:47 |
| *** dsneddon has joined #oooq | 08:51 | |
| *** ykarel is now known as ykarel|lunch | 08:52 | |
| *** jtomasek has joined #oooq | 09:05 | |
| *** dsneddon has quit IRC | 09:05 | |
| *** dsneddon has joined #oooq | 09:06 | |
| marios | ykarel|lunch: was there something more needed after the https://review.openstack.org/#/c/633434/3 (I see scen 1 still skips) | 09:12 |
| *** bogdando has joined #oooq | 09:13 | |
| *** ykarel|lunch is now known as ykarel | 09:20 | |
| ykarel | marios, looking | 09:20 |
| marios | ykarel: e.g. here https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic&job_name=periodic-tripleo-ci-centos-7-scenario004-standalone-master | 09:21 |
| ykarel | marios, now new bug :( | 09:22 |
| ykarel | in container-build | 09:22 |
| marios | ykarel: the cistatus shows them green but they last ran on 25th http://cistatus.tripleo.org/promotion/ | 09:22 |
| marios | ykarel: ack let me know if i can help i'm mostly worrying about having messed up the job definitions but luckily they didn run a bit (on 25th) before skips started | 09:22 |
| ykarel | 1 pass means job is OK | 09:22 |
| marios | ykarel: do we have bug for it | 09:22 |
| ykarel | fix for current issue:- https://review.openstack.org/#/c/633665/ | 09:23 |
| marios | ykarel: ack thanks man looking | 09:23 |
| ykarel | don't think there is a bug though | 09:23 |
| sshnaidm | quiquell, hi | 09:25 |
| ykarel | arxcruz|ruck, FYI ^^ in case u haven't seen container-build Error | 09:25 |
| sshnaidm | quiquell, do you see here error message? It's difficult to understand what went wrong: https://logs.rdoproject.org/39/565839/7/openstack-check/tripleo-ci-reproducer-centos-7-libvirt/d5b9bb5/job-output.txt.gz#_2019-01-28_10_51_45_199824 | 09:25 |
| ykarel | may be file a bug to track | 09:25 |
| marios | ykarel: ack sorry got distracted now looking | 09:26 |
| ykarel | ack | 09:26 |
| marios | arxcruz|ruck: ssbarnea|bkp2 fyi https://review.openstack.org/#/c/633665/ need to unblock master promotion fyi | 09:27 |
| marios | arxcruz|ruck: ssbarnea|bkp2 has no bug ^ | 09:27 |
| *** panda is now known as panda|numb | 09:28 | |
| marios | arxcruz|ruck: ssbarnea|bkp2 ykarel ah but is against openstack/kolla but still may want to track that thanks | 09:30 |
| *** jfrancoa has quit IRC | 09:32 | |
| *** dtantsur|afk is now known as dtantsur | 09:35 | |
| quiquell | sshnaidm: have to fix something so we have journal | 09:36 |
| quiquell | sshnaidm: but that's not virt-resize | 09:36 |
| sshnaidm | quiquell, I mean ansible failure | 09:36 |
| *** derekh has joined #oooq | 09:36 | |
| quiquell | sshnaidm: let's replace that code wit https://docs.ansible.com/ansible/latest/modules/wait_for_connection_module.html | 09:42 |
| quiquell | sshnaidm: so we don't need to delegate | 09:42 |
| sshnaidm | quiquell, do you mean waiting for zuul tenant? | 09:43 |
| quiquell | sshnaidm: nope | 09:44 |
| quiquell | sshnaidm: this is just waiting for libvirt nodes to come up after updating their packages | 09:44 |
| quiquell | sshnaidm: but since we are doing delegate_to maybe the error is not clear | 09:45 |
| sshnaidm | quiquell, ah, ok | 09:45 |
| quiquell | sshnaidm: ansible has a wait_for_connection that just wait until remote is up | 09:45 |
| quiquell | sshnaidm: so we don't have to delegate | 09:45 |
| quiquell | sshnaidm: https://review.rdoproject.org/r/18558 | 09:45 |
| sshnaidm | quiquell, yeah, it's a usual way | 09:45 |
| quiquell | sshnaidm: Also fixed dumping journal | 09:45 |
| sshnaidm | quiquell, you need "become" when dumping journal | 09:47 |
| sshnaidm | quiquell, otherwise you can't see a lot of messages | 09:47 |
| sshnaidm | quiquell, https://logs.rdoproject.org/58/18558/8/check/tripleo-ci-reproducer-centos-7-host/5fd57ce/job-output.txt.gz#_2019-01-29_08_58_58_407963 | 09:48 |
| *** jfrancoa has joined #oooq | 09:49 | |
| quiquell | sshnaidm: done | 09:49 |
| sshnaidm | quiquell, can you rebase also? | 09:49 |
| sshnaidm | quiquell, I don't see a button "rebase", not sure why it isn't there | 09:50 |
| quiquell | sshnaidm: yep same here, something is not right at project config we have to find | 09:50 |
| quiquell | sshnaidm: if you are not the owner the rebase is not there | 09:50 |
| sshnaidm | quiquell, yeah, seems like that | 09:50 |
| quiquell | sshnaidm, ykarel: Do you know where we can get the mirror from zuul cannot find the proper variable https://review.rdoproject.org/r/#/c/18571/ | 09:51 |
| sshnaidm | ykarel, jpena, do you know maybe what is required in gerrit config to see a rebase button? ^^ | 09:51 |
| sshnaidm | quiquell, I suppose it's set by some infra role | 09:53 |
| quiquell | sshnaidm: mirror_fqdn is not and the zuul_foobar neither :-/ | 09:53 |
| quiquell | sshnaidm: maybe they are just abailable at pre | 09:54 |
| arxcruz|ruck | marios: ykarel sorry, i was on the doctor, what's up ? do you want me to open a bug? do you have the log error ? | 09:55 |
| jpena | sshnaidm: there is a "rebase" permission in the resource configuration. Either you are in the owner group, or you have that permission | 09:55 |
| sshnaidm | jpena, thanks | 09:55 |
| jpena | see https://github.com/rdo-infra/review.rdoproject.org-config/blob/8aeb54cd60e70a1cddea0becb2efe9e081f4850b/resources/config.yaml#L38 for an example | 09:55 |
| sshnaidm | quiquell, will do the same ^ | 09:56 |
| marios | arxcruz|ruck: fyi https://review.openstack.org/#/c/633665/ need to unblock master promotion (and has no bug, but is kolla, not sure if we need one , but you should at least be aware ) | 09:56 |
| quiquell | sshnaidm: ack sure go for it | 09:57 |
| arxcruz|ruck | marios: i'll open it and you update the review with the lp ok? | 09:57 |
| marios | arxcruz|ruck: sure but is ykarel review ^ | 09:57 |
| arxcruz|ruck | so ykarel update it :D | 09:57 |
| quiquell | jpena: do you know why this is failing ? https://review.rdoproject.org/r/#/c/18571/ | 09:57 |
| quiquell | jpena: looks like zuul mirror variable is not present at run, maybe just at pre ? | 09:57 |
| *** jfrancoa has quit IRC | 09:59 | |
| jpena | maybe it is because it is set by another role, and we're not importing it at run? | 09:59 |
| *** jfrancoa has joined #oooq | 10:00 | |
| arxcruz|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1813747 | 10:02 |
| openstack | Launchpad bug 1813747 in tripleo "containers-build job is failing with: ERROR:kolla.common.utils:rabbitmq Failed with status: error" [Critical,Triaged] - Assigned to yatin (yatinkarel) | 10:02 |
| arxcruz|ruck | ykarel: i've assigned to you | 10:02 |
| ykarel | arxcruz|ruck, ack | 10:08 |
| *** chem has quit IRC | 10:10 | |
| sshnaidm | quiquell, found only here: https://github.com/openstack-infra/project-config/blob/bc878aedffabd345d6f1c54f59d14cdc60d5f1cd/zuul/site-variables.yaml | 10:11 |
| *** chem has joined #oooq | 10:11 | |
| quiquell | sshnaidm: and RDO? | 10:12 |
| sshnaidm | quiquell, idk, but need to look where "nodepool" is set | 10:15 |
| *** chem has quit IRC | 10:16 | |
| ykarel | sshnaidm, so you have those nodepool vars set https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/31b4cd8/zuul-info/inventory.yaml | 10:18 |
| ykarel | sorry: https://logs.rdoproject.org/71/18571/8/check/tripleo-ci-reproducer-centos-7-libvirt/b7dc8df/zuul-info/inventory.yaml | 10:18 |
| ykarel | sshnaidm, btw any luck with virt-resize issue? | 10:19 |
| sshnaidm | ykarel, not yet afaik :( | 10:19 |
| ykarel | ohhk force-tcg setting didn't helped? | 10:20 |
| *** rf0lc0 has joined #oooq | 10:27 | |
| *** rfolco has quit IRC | 10:27 | |
| quiquell | sshnaidm: it's here https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/files/projects/zuul-config/zuul.d/jobs.yaml#L26 | 10:28 |
| quiquell | sshnaidm: haha this is reproducer :-/ | 10:28 |
| sshnaidm | quiquell, yeah :) it's anyway constructed from nodepool vars | 10:29 |
| quiquell | sshnaidm: cannot find it even at software-factory | 10:30 |
| quiquell | ykarel, jpena: Where do we define zuul_site_mirror_fqdn at RDO zuul ? | 10:30 |
| ykarel | may be we are using NODEPOOL_ vars defined by configure-mirror role | 10:31 |
| sshnaidm | quiquell, I think they use the same openstack-zuul role | 10:31 |
| quiquell | sshnaidm: what role ? | 10:32 |
| chandankumar | quiquell: marios i need some help here https://review.openstack.org/#/c/633185/7/playbooks/multinode-standalone.yml@78 | 10:37 |
| chandankumar | quiquell: in order to get cloud name openstack.clouds will return a list | 10:37 |
| chandankumar | quiquell: then in terms of tripleo ci there will be only one cloudname | 10:37 |
| quiquell | chandankumar: http://www.oznetnerd.com/jinja2-selectattr-filter/ | 10:38 |
| chandankumar | quiquell: in order to get that I need to eject a for loop then look for cloudname then find index | 10:38 |
| quiquell | chandankumar: selectattr help you find a dictionary in a list of dictionaries | 10:38 |
| quiquell | chandankumar: you can specify the attribute name and value you want to match | 10:38 |
| quiquell | chandankumar: "equalto" in your case | 10:39 |
| chandankumar | checking | 10:39 |
| quiquell | | selectattr("name", "equalto", nameofcloud) | 10:39 |
| *** dsneddon has quit IRC | 10:41 | |
| quiquell | ykarel, sshnaidm: found the mirror stuff https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/base/pre.yaml#L5 | 10:42 |
| quiquell | It's local to the play :-/ | 10:42 |
| quiquell | It's just harcoding stuff mirror_fqdn: "mirror.{{ nodepool.region | lower }}.{{ nodepool.cloud | lower }}.rdoproject.org" | 10:43 |
| quiquell | bummer | 10:43 |
| quiquell | Going to do the same | 10:43 |
| sshnaidm | quiquell, I think we have the same in base job, isn't it? | 10:44 |
| quiquell | sshnaidm: noe | 10:44 |
| quiquell | nope | 10:44 |
| quiquell | rdo base jobs does not have that | 10:44 |
| *** chem has joined #oooq | 10:44 | |
| sshnaidm | quiquell, hm.. right | 10:45 |
| quiquell | sshnaidm: but this is not the only job that do docker stuff | 10:46 |
| quiquell | sshnaidm: at RDO how is done by the others ? | 10:46 |
| sshnaidm | quiquell, in tripleo jobs we take it from /etc/ci/mirrors, which is populated by configure-mirrors role | 10:47 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 10:47 |
| sshnaidm | quiquell, and it set mirror_fqdn to "mirror.{{ nodepool.region ... | 10:48 |
| quiquell | sshnaidm: this ? https://github.com/rdo-infra/rdo-jobs/tree/master/roles/mirror-info-fork | 10:48 |
| sshnaidm | quiquell, seems like that | 10:49 |
| quiquell | sshnaidm: that depend on mirror_fqdn XD | 10:49 |
| quiquell | arggg | 10:49 |
| sshnaidm | :D | 10:49 |
| quiquell | sshnaidm: I think we don't have the mirror stuff at RDO jobs | 10:49 |
| sshnaidm | quiquell, in tripleo jobs we do | 10:50 |
| quiquell | sshnaidm: sure ? | 10:50 |
| sshnaidm | quiquell, yeah | 10:50 |
| quiquell | sshnaidm: do you have a trace around ? I think it was a role or something that read that | 10:50 |
| *** rf0lc0 has quit IRC | 10:53 | |
| sshnaidm | quiquell, any ovb job | 10:54 |
| quiquell | sshnaidm: ykarel just show me, it's cooked in the image | 10:55 |
| quiquell | sshnaidm: but with vanilla images we are not going to have that (In case we try to use them | 10:55 |
| quiquell | ) | 10:55 |
| sshnaidm | quiquell, too many channels! :D | 10:55 |
| sshnaidm | quiquell, mirrors is cooked, but environment variables are set in job afaik | 10:56 |
| quiquell | yep sorry | 10:56 |
| quiquell | Nope they are set because we source it at tripleo | 10:56 |
| quiquell | but not per se | 10:56 |
| quiquell | sshnaidm: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/efc8382/job-output.txt.gz#_2019-01-28_22_15_55_603633 | 10:57 |
| sshnaidm | quiquell, and before this: source /etc/nodepool/provider | 10:57 |
| sshnaidm | quiquell, it looks like that: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/efc8382/logs/undercloud/etc/ci/mirror_info.sh.txt.gz | 10:58 |
| quiquell | sshnaidm: but this is toci, we don't use toci for reproducer | 10:58 |
| quiquell | I mean the sourcing | 10:59 |
| sshnaidm | quiquell, yeah | 10:59 |
| quiquell | and vanilla images are not going to have that | 10:59 |
| sshnaidm | quiquell, I mean how it works in tripleo jobs | 10:59 |
| quiquell | sshnaidm: ack | 10:59 |
| sshnaidm | quiquell, but we need to set it for rdo cloud mirror if using rdo cloud | 11:00 |
| sshnaidm | quiquell, so it should be a parameter | 11:00 |
| sshnaidm | quiquell, and then "{% if rdo cloud %}" we set it to rdo cloud mirror, else - our default one | 11:01 |
| *** ssbarnea|bkp2 has quit IRC | 11:01 | |
| quiquell | sshnaidm: ack | 11:02 |
| quiquell | sshnaidm: Let's see first if job is passing now | 11:02 |
| *** ssbarnea|rover has joined #oooq | 11:02 | |
| quiquell | sshnaidm: also we need mirror_fqdn for libvirt and dryrun | 11:03 |
| ssbarnea|rover | hello! i am finally back. | 11:03 |
| *** rfolco has joined #oooq | 11:03 | |
| quiquell | ssbarnea|rover: hey there welcome back | 11:03 |
| *** dsneddon has joined #oooq | 11:06 | |
| sshnaidm | quiquell, yeah, and for this too: https://tree.taiga.io/project/tripleo-ci-board/task/398 | 11:06 |
| quiquell | sshnaidm: so would be nice to run repro CI at less cooked images | 11:07 |
| quiquell | sshnaidm: for that I am doing this https://review.rdoproject.org/r/#/c/18593/ | 11:07 |
| quiquell | sshnaidm: And that image does not have the mirror info to source | 11:07 |
| quiquell | sshnaidm: so we need to configure docker mirroring ourself, one option is to just add openstack and rdo to the list | 11:08 |
| quiquell | sshnaidm: if RDO is down it will use openstack | 11:08 |
| sshnaidm | quiquell, rdo should use only rdo, all the rest - openstack | 11:08 |
| *** udesale has quit IRC | 11:09 | |
| sshnaidm | quiquell, rdo mirrors are not accessible not from rdo cloud | 11:09 |
| bogdando | ykarel: hi. Thanks for feedback for https://review.openstack.org/#/c/633484 | 11:09 |
| bogdando | I think I'm stuck there tho | 11:09 |
| bogdando | PTAL for the latest comments | 11:09 |
| bogdando | folks ^^ | 11:09 |
| sshnaidm | quiquell, openstack mirror is accessible from all I think.. but in rdo cloud it'll take hours | 11:09 |
| quiquell | sshnaidm: ack, then let's just do the switch at job definition | 11:09 |
| quiquell | sshnaidm: so mirror_fqdn is accessible to all reproducer jobs | 11:10 |
| sshnaidm | quiquell, I think we can add it to cloud variables we have right now.. | 11:10 |
| quiquell | sshnaidm: what variables ? | 11:10 |
| ykarel | bogdando, reading | 11:10 |
| bogdando | ykarel: did you mean perhaps that just amending the available var with --provides covers it fully? | 11:10 |
| bogdando | I think I cna get that... so we can just place old names to the list so they as well end up into the final list | 11:11 |
| bogdando | neat | 11:11 |
| ykarel | bogdando, yes | 11:11 |
| bogdando | okay | 11:11 |
| *** dsneddon has quit IRC | 11:11 | |
| quiquell | sshnaidm: Wait for nodes failing again http://logs.rdoproject.org/58/18558/12/check/tripleo-ci-reproducer-centos-7-libvirt/725cc50/job-output.txt.gz | 11:11 |
| ykarel | bogdando, the command i shared will cover the cases mentioned in bug | 11:11 |
| quiquell | sshnaidm: I think we have to dimension timeouts and the like | 11:11 |
| bogdando | your repoquery kungfu is great, sansay ykarel | 11:11 |
| sshnaidm | quiquell, somewhere here maybe: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/defaults/main.yaml#L27-L33 | 11:12 |
| quiquell | sshnaidm: those variables are not seen by zuul job | 11:13 |
| sshnaidm | quiquell, as I see in logs it didn't wait at all | 11:13 |
| sshnaidm | quiquell, I think we can pass them to base job | 11:13 |
| sshnaidm | quiquell, or templatize base job | 11:14 |
| quiquell | sshnaidm: yep it's not waiting.. :-/ | 11:14 |
| quiquell | sshnaidm: wait I see an error before | 11:14 |
| sshnaidm | quiquell, maybe we need start to stabilize what we have now, including mirrors adding.. not too much time till sprint end | 11:15 |
| quiquell | sshnaidm: "grep: /etc/ssh/ssh_known_hosts: No such file or directory" | 11:15 |
| quiquell | sshnaidm: why I cannot link to line at logs ? maybe we are missing a post role or something ? | 11:15 |
| sshnaidm | quiquell, why can not? You can | 11:16 |
| sshnaidm | quiquell, like that? https://logs.rdoproject.org/58/18558/12/check/tripleo-ci-reproducer-centos-7-libvirt/725cc50/job-output.txt.gz#_2019-01-29_10_52_03_869200 | 11:16 |
| quiquell | sshnaidm: can you do that here ? http://logs.rdoproject.org/58/18558/12/check/tripleo-ci-reproducer-centos-7-libvirt/725cc50/job-output.txt.gz | 11:17 |
| quiquell | sshnaidm: I can't | 11:17 |
| sshnaidm | quiquell, click on timstamp | 11:17 |
| ssbarnea|rover | who is working on the new reproducer? i found https://logs.rdoproject.org/39/565839/7/openstack-check/tripleo-ci-reproducer-centos-7-libvirt/d5b9bb5/job-output.txt.gz#_2019-01-28_10_51_44_993395 | 11:17 |
| quiquell | sshnaidm: Is not clickable for me | 11:17 |
| sshnaidm | quiquell, hmm.. weird, ctrl+f5? | 11:18 |
| sshnaidm | ssbarnea|rover, it's more experimental stuff, can fail in some places | 11:18 |
| ssbarnea|rover | panda|numb: do you have few minutes? | 11:19 |
| panda|numb | ssbarnea|rover: I'm numb | 11:19 |
| quiquell | ssbarnea|rover: Yep, I think this is the fatal stuff | 11:20 |
| quiquell | sshnaidm: ^ the problem for libvirt is there | 11:20 |
| quiquell | Have to figure out what it's not fail fasting there | 11:21 |
| quiquell | so subnode-0 has the issue but subnode-1 is working fine :-/ | 11:22 |
| quiquell | sshnaidm: so the issue it's a subnode-0 I use a playbook to be able to do that in parallel | 11:23 |
| quiquell | ssbarnea|rover: but it's more difficult to debug | 11:23 |
| quiquell | ss argg | 11:24 |
| ssbarnea|rover | quiquell: my impression is that this could have being avoided if we would respect E405 rule (ansible-lint). https://docs.ansible.com/ansible-lint/rules/default_rules.html --- current code is *not* compliant. | 11:26 |
| ssbarnea|rover | @oooq: more votes on https://review.openstack.org/#/c/632695/ needed? (documenting use of @oooq tag). thanks. | 11:30 |
| quiquell | ssbarnea|rover: But we want to install latest there | 11:34 |
| quiquell | ssbarnea|rover: It was like that at tqe | 11:34 |
| quiquell | ssbarnea|rover: going to remove latest and see what happend, thanks | 11:34 |
| quiquell | ssbarnea|rover: also later on we do an update | 11:35 |
| quiquell | ssbarnea|rover: how do we update the system and respect E405 rule ? | 11:35 |
| ssbarnea|rover | quiquell: add a retry to package module call. 2-3 retries are more than enough. | 11:36 |
| quiquell | Ahh silly me is about retries | 11:36 |
| quiquell | ssbarnea|rover: the issue was not that | 11:36 |
| quiquell | ssbarnea|rover: issue was ansible lint states that we cannot use "latest" | 11:36 |
| ssbarnea|rover | quiquell: do not use the infitite loop example from the docs, is... unfortunate ;) | 11:36 |
| quiquell | ssbarnea|rover: and what about the state: latest ? | 11:37 |
| quiquell | ssbarnea|rover: let me reproduce the ansible-lint stuff | 11:37 |
| quiquell | ssbarnea|rover: so you can help me there | 11:37 |
| ssbarnea|rover | quiquell: this is ok | 11:37 |
| ssbarnea|rover | quiquell: i doubt we will ever not use this on CI. | 11:37 |
| quiquell | ssbarnea|rover: tasks/libvirt/prepare.yaml:32: [E403] Package installs should not use latest | 11:37 |
| quiquell | ssbarnea|rover: that's the error I didn't want | 11:37 |
| ssbarnea|rover | quiquell: add this: # noqa: E403 to the line. | 11:38 |
| quiquell | ssbarnea|rover: cool! that's the stuff I was looking for | 11:38 |
| quiquell | thanks! | 11:38 |
| ssbarnea|rover | newer versions of ansible-lint do know about the same "noqa" comments, which is great, no more need to use skip_ansible_lint | 11:38 |
| quiquell | ssbarnea|rover: Do I have to update the pre-commit stuff ? | 11:39 |
| quiquell | To get new ansible-lint ? | 11:39 |
| quiquell | ssbarnea|rover: not working | 11:39 |
| ssbarnea|rover | quiquell: on some repos you do have to: "pre-commit autoupdate" | 11:39 |
| quiquell | ssbarnea|rover: did that | 11:40 |
| quiquell | + # noqa: E403 to the line. | 11:40 |
| quiquell | state: latest | 11:40 |
| quiquell | This is it? | 11:40 |
| marios | biab | 11:40 |
| ssbarnea|rover | i think this was done in 4.0.1, let me check. not sure about exact syntax. | 11:40 |
| *** dsneddon has joined #oooq | 11:40 | |
| quiquell | ssbarnea|rover: btw it does not complain of retries | 11:40 |
| ssbarnea|rover | quiquell: i know, because this rule is disabled in .ansible-lint - its use is a big controversial, even inside our team. we need to check with others if we are about to accept and respect it or not, or if we will only add retries where they are proven to be needed. | 11:43 |
| ssbarnea|rover | quiquell: we should also watch https://github.com/ansible/ansible-lint/issues/456 -- to see in which direction it goes. | 11:44 |
| quiquell | ssbarnea|rover: man cannot make the noqa thing working | 11:44 |
| ssbarnea|rover | it seems that E405 will no longer be enabled by default in 4.1.0, not really surprised. | 11:44 |
| quiquell | ssbarnea|rover: this is it? | 11:45 |
| quiquell | https://github.com/ansible/ansible-lint/pull/460/files | 11:45 |
| *** dsneddon has quit IRC | 11:45 | |
| quiquell | ssbarnea|rover: could the retry fix the issue ? | 11:45 |
| ssbarnea|rover | quiquell: now i know why you cannot use noqa: it was not released yet. it was merged into master 19 days ago but latest release is 25 days old. | 11:45 |
| quiquell | ssbarnea|rover: arg, ok I am going to keep the skip | 11:46 |
| ssbarnea|rover | quiquell: i believe so. I would put a retry there. cannot hurt and had nice benefits: we can even measure how often retries are used to salvage builds using logstash. | 11:46 |
| quiquell | ssbarnea|rover: Let's do that at both package install at prepare.yaml | 11:47 |
| quiquell | sshnaidm: we have failing hypervisor http://logs.rdoproject.org/58/18558/13/check/tripleo-ci-reproducer-centos-7-libvirt/8f90930/job-output.txt.gz | 11:47 |
| quiquell | ykarel: ^ thanks | 11:48 |
| ssbarnea|rover | quiquell: cc me on review. i am for it. the 3 retry lines could make our builds more resilient to random networking issues. | 11:48 |
| quiquell | argg sh: getenforce: command not found | 11:48 |
| ykarel | quiquell, i can't find full logs, where did u see it's hypervisor issue | 11:51 |
| quiquell | ykarel: na forget we cannot run the tool at centos7 | 11:51 |
| quiquell | sh: getenforce: command not found | 11:51 |
| ykarel | quiquell, not sure if that's related to hypervisor | 11:52 |
| quiquell | ykarel: Is not | 11:52 |
| quiquell | ykarel: the thing is that we are trying to run libguestfs-test-tool at centos7 in CI | 11:52 |
| quiquell | ykarel: And it fails because it expect selinux there | 11:52 |
| ykarel | that command comes from: libselinux-utils | 11:53 |
| ykarel | it may be it's not in PATH | 11:53 |
| *** holser_ has joined #oooq | 11:55 | |
| quiquell | ssbarnea|rover: just added 5 retries at both places, let's see if it helps | 11:55 |
| quiquell | sshnaidm: ack on stabilizing I am throwing random reviews at reproducer right now | 11:56 |
| quiquell | sshnaidm: let's merge the finger server https://review.rdoproject.org/r/#/c/18475/ | 11:57 |
| sshnaidm | quiquell, commented | 11:59 |
| quiquell | sshnaidm: going to add logging file too to the finger service | 12:06 |
| quiquell | sshnaidm: https://review.rdoproject.org/r/18475 | 12:06 |
| sshnaidm | quiquell, port 79? | 12:10 |
| sshnaidm | quiquell, no problem with no-root user? | 12:10 |
| quiquell | yep | 12:10 |
| quiquell | noe | 12:10 |
| quiquell | nope | 12:10 |
| quiquell | weird ehh | 12:10 |
| quiquell | sshnaidm: so were do we put the mirror stuff ? | 12:11 |
| quiquell | sshnaidm: like conditional in the job ? | 12:11 |
| *** dsneddon has joined #oooq | 12:12 | |
| *** agopi|off has joined #oooq | 12:12 | |
| sshnaidm | quiquell, it's used in pre.yml of base job, right? | 12:12 |
| quiquell | sshnaidm: nope in run.yaml I don't want to pollute pre.yaml | 12:13 |
| sshnaidm | quiquell, https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/files/projects/zuul-config/playbooks/base/pre.yaml#L27 | 12:13 |
| quiquell | sshnaidm: run.yaml is pure zuul | 12:13 |
| quiquell | sshnaidm: I am talking about CI not CI(CI) | 12:13 |
| sshnaidm | quiquell, I see | 12:14 |
| sshnaidm | confusing :D | 12:14 |
| quiquell | sshnaidm: so this is working | 12:14 |
| quiquell | https://review.rdoproject.org/r/#/c/18571/ | 12:14 |
| quiquell | sshnaidm: just harcode the RDO mirror in the job defintion | 12:14 |
| quiquell | sshnaidm: but running this job out of CI will be not good so we have to able to switch that | 12:15 |
| *** jpena is now known as jpena|lunch | 12:15 | |
| quiquell | sshnaidm: I can look at zuul_mirror_fqdn if not present set RDO | 12:15 |
| quiquell | sshnaidm: and we set zuul_mirrof_fqdn at repro startup in zuul | 12:16 |
| quiquell | sshnaidm: what do you think ? | 12:16 |
| *** dsneddon has quit IRC | 12:16 | |
| sshnaidm | quiquell, I think it should be depending on user config of cloud, let's no rely on presence of zuul_mirror_fqdn | 12:16 |
| quiquell | sshnaidm: but we don't have that in CI | 12:17 |
| sshnaidm | quiquell, which CI? | 12:17 |
| quiquell | sshnaidm: RDO CI | 12:17 |
| quiquell | sshnaidm: not CI(CI) | 12:18 |
| quiquell | RDO(CI) | 12:18 |
| sshnaidm | quiquell, let's sync, I'm total lost | 12:18 |
| quiquell | blue ? | 12:18 |
| quiquell | going to yours | 12:18 |
| *** skramaja has quit IRC | 12:22 | |
| *** ratailor has quit IRC | 12:31 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 12:47 |
| *** quiquell is now known as quiquell|brb | 12:48 | |
| *** dsneddon has joined #oooq | 12:49 | |
| quiquell|brb | sshnaidm: finger service fixed https://review.rdoproject.org/r/#/c/18475/ | 12:51 |
| sshnaidm | quiquell|brb, I remove this change, it's not related: https://review.rdoproject.org/r/#/c/18571/13/playbooks/tripleo-ci-reproducer/pre.yaml | 12:52 |
| *** apetrich has quit IRC | 12:52 | |
| quiquell|brb | ack | 12:52 |
| quiquell|brb | sshnaidm: done, going back in a few | 12:54 |
| *** apetrich has joined #oooq | 12:55 | |
| *** agopi|off has quit IRC | 12:55 | |
| *** dsneddon has quit IRC | 12:55 | |
| chandankumar | quiquell|brb: https://review.openstack.org/#/c/633185/8/playbooks/multinode-standalone.yml@78 it is not working here I am not sure I have done something wrong here | 12:59 |
| weshay | is rdo back | 12:59 |
| weshay | arxcruz|ruck, ssbarnea|rover ? | 12:59 |
| weshay | chandankumar, arxcruz|ruck kopecmartin https://bluejeans.com/u/whayutin/ | 13:00 |
| weshay | arxcruz|ruck, hola hola? | 13:02 |
| arxcruz|ruck | weshay: joining | 13:04 |
| *** trown|outtypewww is now known as trown | 13:13 | |
| *** gkadam has quit IRC | 13:20 | |
| panda|numb | quiquell|brb: where are zuul containers logging ? | 13:25 |
| *** quiquell|brb is now known as quiquell | 13:26 | |
| quiquell | panda|numb: docker-compose logs | 13:26 |
| *** agopi|off has joined #oooq | 13:26 | |
| *** agopi|off is now known as agopi | 13:26 | |
| ssbarnea|rover | weshay: yeah, back | 13:29 |
| quiquell | weshay: Looks like we have a less coocked images for rdo jobs https://review.rdoproject.org/r/#/c/18593/ | 13:29 |
| quiquell | weshay: so we can find issues with pre.yaml | 13:29 |
| *** rlandy has joined #oooq | 13:30 | |
| weshay | k k.. I want to catch up but chatting w/ tempest guys | 13:30 |
| quiquell | This to have it for fedora28https://softwarefactory-project.io/r/#/c/14897/ | 13:30 |
| quiquell | weshay: ack, rebase your review in top of that so you can check | 13:30 |
| chandankumar | weshay: https://review.openstack.org/#/c/589568 | 13:30 |
| *** dsneddon has joined #oooq | 13:30 | |
| weshay | ssbarnea|rover, https://docs.google.com/document/d/1Zyzez6YStxEvXs8cBo6RetoigOy9E68UNc9rfXBSY3Y/edit?ts=5c504a27 | 13:31 |
| chandankumar | weshay: https://review.openstack.org/#/c/581214/ | 13:31 |
| quiquell | rlandy: let's merge docker RDO mirror https://review.rdoproject.org/r/#/c/18571/ | 13:33 |
| quiquell | rlandy: working now we are being hit by cloudflare TLS issues | 13:33 |
| rlandy | quiquell: k- will do | 13:33 |
| quiquell | rlandy: about libvirt looks like some hypervisors have a bug at centos | 13:34 |
| quiquell | rlandy: fedora is not affected | 13:34 |
| rlandy | quiquell: my jobs last night didn't launch | 13:34 |
| rlandy | but got as far as that | 13:34 |
| rlandy | quiquell: any other reviews that need to merge? | 13:35 |
| *** jpena|lunch is now known as jpena | 13:36 | |
| quiquell | rlandy: https://review.rdoproject.org/r/#/c/18571/ | 13:37 |
| *** dsneddon has quit IRC | 13:38 | |
| quiquell | rlandy: this is important too but I am fixing it | 13:38 |
| quiquell | rlandy: https://review.rdoproject.org/r/#/c/18569/ | 13:38 |
| rlandy | top one is done | 13:39 |
| quiquell | rlandy: with the top one now we can show how to stream console logs at reproducer | 13:39 |
| rlandy | quiquell: k - going to recheck one before | 13:39 |
| quiquell | rlandy: ack will try to fix this | 13:40 |
| rlandy | k - will keep an eye on it | 13:40 |
| *** matbu has quit IRC | 13:42 | |
| rlandy | quiquell: https://review.rdoproject.org/r/#/c/18606/ - very minor change - but job won't start if someone copies this example otherwise | 13:47 |
| rlandy | weshay: wrt https://review.openstack.org/#/c/631067/30/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 - you wanted docker install added and what else? | 13:48 |
| quiquell | rlandy: +w | 13:48 |
| marios | so this system eng all hands is at same time as tripleo weekly | 13:53 |
| *** matbu has joined #oooq | 13:55 | |
| panda|numb | marios: ? | 13:56 |
| *** panda|numb is now known as panda | 13:56 | |
| panda | still kind of numb | 13:56 |
| marios | in my calendar panda (System Engineering All hands ) | 13:56 |
| panda | marios: why ? you want to attend ? | 13:56 |
| marios | don't know should we | 13:56 |
| marios | i answere maybe cos wasn't sure :) | 13:57 |
| panda | are we system engineers ? | 13:57 |
| marios | panda: radio system-end | 13:58 |
| marios | eng even | 13:58 |
| chandankumar | weshay: barbican stuff https://review.openstack.org/#/c/631326/ | 13:59 |
| *** ykarel is now known as ykarel|away | 14:01 | |
| *** dsneddon has joined #oooq | 14:02 | |
| ssbarnea|rover | sshnaidm: weshay: please help me merge https://review.rdoproject.org/r/#/c/18294/ -- correction of ovb repo location for ci-config. | 14:05 |
| sshnaidm | ssbarnea|rover, it's obsolete | 14:06 |
| weshay | ssbarnea|rover, te-broker is dead dude | 14:06 |
| ssbarnea|rover | weshay: te-broker is the name of the *machine*... | 14:06 |
| sshnaidm | ssbarnea|rover, machine is live, but service is dead and unused | 14:06 |
| *** dsneddon has quit IRC | 14:06 | |
| ssbarnea|rover | sshnaidm: nope,the cleanup script is deployed by this script, so it is used. | 14:07 |
| weshay | right.. so the right thing would be to remove some of this setup.. and maybe add something to ensure the cleanup script is on it and running | 14:07 |
| weshay | if it's not already written | 14:07 |
| ssbarnea|rover | weshay++ | 14:07 |
| hubbot1 | ssbarnea|rover: weshay's karma is now 14 | 14:07 |
| ssbarnea|rover | weshay: this is the reason why i was working on this, for the cleanup script. | 14:08 |
| weshay | k | 14:08 |
| weshay | it's not obvious from the patch | 14:08 |
| weshay | can you update the commit message, then we'll merge.. | 14:08 |
| ssbarnea|rover | weshay: just to double check: the only thing to be kept out of te-broker setup is the cron part, right? i can remove other stuff from the playbook/role. | 14:08 |
| weshay | not sure why we need ovb repo for the cleanup | 14:09 |
| ssbarnea|rover | weshay: well, i can remove it and this will resolve the issue. doing it right now. | 14:10 |
| weshay | rlandy, re: https://review.openstack.org/#/c/631067/30/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 14:11 |
| quiquell | rlandy: about starting reproducer without zuul restart it's "--skip-tags start" | 14:11 |
| weshay | I was going to clean up and toy around w/ installing docker and setting up the user group in the script | 14:11 |
| rlandy | weshay: sure - I am working on some basic user doc now | 14:12 |
| rlandy | so we have something to instruct the user what to do with these files | 14:12 |
| rlandy | quiquell: thanks | 14:12 |
| rlandy | quiquell: and finally the tar file you suggested ages ago ... http://logs.openstack.org/67/631067/30/check/tripleo-ci-centos-7-undercloud-containers/7c217a0/logs/reproducer-zuul-based-quickstart.tar | 14:13 |
| ssbarnea|rover | weshay: i guess it could be wise to rename the te-broker machie/role/playbook to something else, to avoid confusions. any ideas? | 14:15 |
| weshay | tripleo-ci-tools maybe? | 14:15 |
| ssbarnea|rover | "tooler"? (shorter) | 14:16 |
| weshay | ssbarnea|rover, raise it in a planning mtg.. <--- panda | 14:16 |
| quiquell | rlandy: We get a tarball so people can decide on runing the script or the playbook, is that right ? | 14:17 |
| quiquell | rlandy: maybe we can make the tarball executable | 14:18 |
| rlandy | quiquell: ack - exactly | 14:18 |
| quiquell | rlandy: https://www.linuxjournal.com/node/1005818 | 14:18 |
| weshay | folks https://docs.google.com/document/d/1yn3SW0-UzAFvBk4xS672hZmNHjFeVo7ag8O2FPpA5bM/edit | 14:18 |
| rfolco | CI community meeting (office hours) starts now at https://bluejeans.com/4113567798 - agenda: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:18 |
| quiquell | rlandy: we can make the thing slef stracting and run the repro bash script | 14:18 |
| weshay | rfolco, there is an all hands atm | 14:19 |
| weshay | so people may be joining late | 14:19 |
| panda | weshay: ssbarnea|rover added to retro https://trello.com/c/XIhded2W/774-rename-te-broker | 14:19 |
| rfolco | weshay, k | 14:19 |
| rlandy | quiquell: extracting is good - running not sure we want to do that - as the bash script can run with various options | 14:19 |
| rlandy | will look | 14:20 |
| quiquell | rlandy: You can put the frontier where you want just extracting or running the script | 14:20 |
| quiquell | rlandy: Just take a look at check if it can help | 14:20 |
| rlandy | ack | 14:20 |
| ssbarnea|rover | ok, not sure if we need much discussion/planning about such trivial issues like renaming a role. | 14:20 |
| quiquell | rlandy: another one, for non libvirt reproducer we don't want to clone tq | 14:22 |
| marios | rfolco: o/ i am listening in the systems eng call. i'll join community if theres stuff in the agenda (watching) | 14:22 |
| rlandy | quiquell: we discussed this at yesterday's meeting - where I mentioned you comment to not install tq when not running libvirt | 14:22 |
| rlandy | quiquell: weshay's suggestion was that we should stick to what ci does | 14:23 |
| rlandy | tq is installed bu ci whether it is used or not | 14:23 |
| rfolco | marios, ack | 14:23 |
| rlandy | making the switch is no big deal | 14:23 |
| rlandy | I am open to either | 14:23 |
| panda | marios: anything interesting ? | 14:23 |
| *** udesale has joined #oooq | 14:24 | |
| rlandy | quiquell: If you feel strongly about it, feel fre eto object and I'll change it | 14:24 |
| marios | panda: yah i mean not uninteresting anyway :) | 14:24 |
| *** ykarel|away has quit IRC | 14:25 | |
| *** holser__ has joined #oooq | 14:26 | |
| *** holser_ has quit IRC | 14:26 | |
| quiquell | rlandy: It's not an stopper but the less we do the better | 14:27 |
| rlandy | quiquell: k , understood - adding the extracting stuff | 14:28 |
| quiquell | rlandy: ack, nice | 14:28 |
| ssbarnea|rover | weshay: panda: ovb-cleanup script fix and tested in production: https://review.rdoproject.org/r/#/c/18517/ see https://gist.github.com/ssbarnea/1e463bad540352e54a817aa876ea6b63 | 14:33 |
| ssbarnea|rover | i manually deployed it to te-broker to be sure that it works. You can check its output running: journalctl -t ovb-tenant-cleanup | 14:34 |
| ssbarnea|rover | panda recommended use of journalctl instead of log files. | 14:34 |
| marios | quiquell: so fedora-28-libvirt green now /me checks logs https://review.rdoproject.org/r/#/c/18539/ | 14:36 |
| quiquell | marios: dryrun ? | 14:36 |
| marios | quiquell: well yeah i'm passing it | 14:37 |
| marios | quiquell: no longer fails mirror at least http://logs.rdoproject.org/39/18539/4/check/tripleo-ci-reproducer-fedora-28-libvirt/8c0f8a0/job-output.txt.gz | 14:38 |
| marios | quiquell: oh wrong log | 14:38 |
| marios | quiquell: sec | 14:38 |
| marios | quiquell: whichi reminds me can we do something about those directories 01/1001/101/ etc | 14:39 |
| quiquell | marios: looks like there is one retry | 14:39 |
| quiquell | marios: weird | 14:39 |
| marios | quiquell: http://logs.rdoproject.org/39/18539/4/check/tripleo-ci-reproducer-fedora-28-libvirt/8c0f8a0/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-reprozuul-dryrun/7f0cd1c/job-output.txt.gz | 14:39 |
| quiquell | marios: I think we can change the post stuff in the base job | 14:39 |
| quiquell | marios: but you can be running a lot of sh.. at your zuul so it's good to differenciate | 14:40 |
| quiquell | marios: is working fine ! \o/ | 14:41 |
| marios | quiquell: well i can't see any error it that nested output log | 14:41 |
| marios | quiquell: can we remove /logs/01/1001/1 directories to something more predictable ;) | 14:41 |
| *** dsneddon has joined #oooq | 14:42 | |
| quiquell | marios: interesting the host is just failing because the RDO image is different from the image we prepare at libvirt | 14:42 |
| quiquell | :-) | 14:42 |
| *** weshay has quit IRC | 14:43 | |
| quiquell | marios: but jobs are running at RDO, what can be the difference | 14:43 |
| quiquell | marios: what we can do is create a symbolic link to latest logs | 14:44 |
| quiquell | logs/latest | 14:44 |
| quiquell | or the like | 14:44 |
| quiquell | But it can have concurrent issues | 14:44 |
| marios | quiquell: ack but what do we want for the sprint? can we merge anything here. or keep this for next sprint (tomorrow) | 14:45 |
| ssbarnea|rover | feel free to show your interest about seeing https://github.com/ansible/proposals/issues/92 fixed. | 14:46 |
| *** dsneddon has quit IRC | 14:46 | |
| quiquell | marios: btw looks like we can set vars without new job definition | 14:47 |
| quiquell | marios: so for the moment we can do dryrun only at libvirt | 14:47 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 14:47 |
| marios | quiquell: adding some status in https://tree.taiga.io/project/tripleo-ci-board/task/615 for end of sprint anyway | 14:47 |
| quiquell | marios: centos libvirt is docker cloudflare stuff there is a fixing merging | 14:50 |
| quiquell | marios: maybe you can help me review this, don't know what is wrong have tested locally https://review.rdoproject.org/r/#/c/18569 | 14:52 |
| marios | quiquell: ack will have a look in a bit | 14:53 |
| marios | quiquell: ah ack (its the hard coded id rsa thing ) | 14:55 |
| quiquell | marios: Don't know the issue and zuul output is not helping much :-( | 14:55 |
| marios | quiquell: wonder if its the user | 14:55 |
| quiquell | ansible_user right ? | 14:55 |
| marios | quiquell: like you're using ansible_user_dir | 14:55 |
| quiquell | sometimes is not there | 14:55 |
| marios | quiquell: yeah i wonder | 14:56 |
| quiquell | ansible_user_dir it's always there in zuul I think | 14:56 |
| quiquell | sshnaidm: legit centos7 failing at libvirt with dumping and all http://logs.rdoproject.org/58/18558/18/check/tripleo-ci-reproducer-centos-7-libvirt/5988a09/ | 14:56 |
| quiquell | sshnaidm: libguestfs failing as expected | 14:56 |
| marios | quiquell: will have a closer look on fresh coffee brain tomorrow | 14:58 |
| sshnaidm | quiquell, why does it fail? | 14:59 |
| sshnaidm | quiquell, can't find the error.. | 14:59 |
| quiquell | sshnaidm: test tool fails but don't know why | 14:59 |
| sshnaidm | quiquell, yeah, me too :) | 14:59 |
| quiquell | sshnaidm: at least we can detect it, then we have to do the fallback to force_cgf (or whatever the name is) but this get sloww | 15:00 |
| sshnaidm | quiquell, how much slower? | 15:00 |
| quiquell | sshnaidm: Well I think we where having it by default somewhere ykarel knows better | 15:00 |
| quiquell | sshnaidm: let's try it | 15:00 |
| quiquell | sshnaidm: what was the name force_fgc? | 15:01 |
| sshnaidm | quiquell, tgc..? | 15:01 |
| quiquell | sshnaidm: Let's merge the dumping at least is a fail fast instead of timeout | 15:03 |
| quiquell | sshnaidm: what do you think ? | 15:03 |
| sshnaidm | quiquell, yeah, but not sure it represents good.. | 15:04 |
| quiquell | what do you mean ? | 15:04 |
| *** sshnaidm is now known as sshnaidm|mtg | 15:04 | |
| chandankumar | quiquell: Hello | 15:07 |
| chandankumar | quiquell: https://review.openstack.org/#/c/633185/8/playbooks/multinode-standalone.yml@79 | 15:09 |
| chandankumar | quiquell: this one is not working as it is coming undefined | 15:09 |
| chandankumar | quiquell: please have a look | 15:10 |
| quiquell | chandankumar: checking | 15:14 |
| quiquell | chandankumar: have you try it at your machine with a cloud.yaml and something similar a playbook ? | 15:14 |
| *** dsneddon has joined #oooq | 15:17 | |
| quiquell | chandankumar: commented | 15:18 |
| *** udesale has quit IRC | 15:18 | |
| quiquell | chandankumar: ahh wait | 15:20 |
| sshnaidm|mtg | arxcruz|ruck, do we have live migration test in tempest? | 15:20 |
| arxcruz|ruck | sshnaidm|mtg: yes, but it not work unless you have two compute nodes | 15:21 |
| sshnaidm|mtg | arxcruz|ruck, ack | 15:21 |
| quiquell | chandankumar: you can filter at the gathering stuff | 15:23 |
| quiquell | chandankumar: for example | 15:23 |
| quiquell | os_client_config: | 15:23 |
| quiquell | clouds: | 15:23 |
| quiquell | - rdo-cloud | 15:23 |
| *** dsneddon has quit IRC | 15:24 | |
| quiquell | chandankumar: commented with right solution | 15:26 |
| chandankumar | quiquell: sure thanks , i will take a look | 15:26 |
| rlandy | sshnaidm|mtg: ok - fixed my python2 error | 15:28 |
| *** sshnaidm|mtg is now known as sshnaidm | 15:33 | |
| *** fultonj has joined #oooq | 15:35 | |
| sshnaidm | rlandy, great | 15:37 |
| quiquell | rlandy: to have the stream command after geting UUID https://review.rdoproject.org/r/#/c/18613/ | 15:37 |
| quiquell | sshnaidm: ^ | 15:37 |
| *** fultonj has joined #oooq | 15:37 | |
| rlandy | k | 15:38 |
| quiquell | sshnaidm: This will work ? https://review.rdoproject.org/r/#/c/18558/20/playbooks/tripleo-ci-reproducer/templates/run.sh.j2 | 15:38 |
| quiquell | sshnaidm: I mean this will affect the libvirt roles ? | 15:38 |
| quiquell | sshnaidm: or they have like a clean environment ? | 15:38 |
| *** ykarel|away has joined #oooq | 15:38 | |
| sshnaidm | quiquell, I don't think it will work tbh | 15:39 |
| *** ykarel|away is now known as ykarel | 15:39 | |
| quiquell | sshnaidm: It has to be passed to the role ? | 15:39 |
| sshnaidm | quiquell, it should be env var for task | 15:39 |
| sshnaidm | quiquell, I think for task/block, not sure about playbook though | 15:39 |
| sshnaidm | quiquell, each task is separate shell | 15:40 |
| quiquell | sshnaidm: then we have to do this just for the virt-size | 15:40 |
| sshnaidm | quiquell, but we have this as a parameter in oooq roles, isn't it? | 15:40 |
| quiquell | sshnaidm: playbook not role, I am not using playbooks | 15:41 |
| sshnaidm | quiquell, then need to specify for each task | 15:41 |
| rlandy | so we still need: sudo dnf install pipenv git ansible-python3 python3-libselinux? | 15:42 |
| quiquell | rlandy: pre.yaml have all this | 15:42 |
| quiquell | rlandy: pipenv is not longer needed we are just using pip --user | 15:42 |
| rlandy | good - removing | 15:42 |
| quiquell | rlandy: no virtualenv or nothing | 15:42 |
| rlandy | quiquell: how about sudo curl -L -o /etc/yum.repos.d/delorean.repo http://trunk.rdoproject.org/fedora/current/delorean.repo | 15:42 |
| rlandy | sudo dnf update -y | 15:42 |
| quiquell | rlandy: same you have at the reproducer scripts | 15:42 |
| rlandy | still need dlrn repos? | 15:42 |
| quiquell | rlandy: dlrn repos ? | 15:43 |
| quiquell | rlandy: in the host ? | 15:43 |
| sshnaidm | rlandy, why do we need dlrn repos? | 15:43 |
| rlandy | quiquell: so the doc says | 15:43 |
| ykarel | sshnaidm, quiquell it's force_tcg | 15:43 |
| sshnaidm | ykarel, ack | 15:43 |
| rlandy | https://docs.google.com/document/d/1i1-_L2mx8_erETAVblchLnymC6-9h7A2x02begy99UQ/edit# | 15:43 |
| quiquell | ykarel: I have to be set per task isn't it ? | 15:43 |
| ykarel | sshnaidm, quiquell can do like http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/modify-image/tasks/libguestfs.yml#n50 | 15:43 |
| quiquell | ykarel: was just checkint it | 15:44 |
| ykarel | in libvirt-nodepool rool | 15:44 |
| ykarel | role | 15:44 |
| rlandy | quiquell: just putting on some user doc to go in the README in each job | 15:44 |
| rlandy | basic stuff | 15:44 |
| quiquell | rlandy: so pre.yaml have all they need I think | 15:44 |
| quiquell | rlandy: if not we change pre.yaml | 15:44 |
| sshnaidm | rlandy, seems like very outdated doc | 15:44 |
| quiquell | rlandy: like the stuff is doing wes | 15:44 |
| rlandy | great - moving on | 15:44 |
| *** ccamacho has quit IRC | 15:45 | |
| *** ccamacho has joined #oooq | 15:46 | |
| quiquell | ykarel: like this https://review.openstack.org/633444 ? | 15:48 |
| quiquell | sshnaidm: ^ | 15:49 |
| *** weshay has joined #oooq | 15:49 | |
| sshnaidm | quiquell, yep, let's see how it goes | 15:50 |
| quiquell | sshnaidm: rolling | 15:50 |
| ykarel | quiquell, Why not doing at http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/playbooks/libvirt-nodepool.yml#n26 | 15:50 |
| ykarel | isn't ^^ used? | 15:51 |
| quiquell | ykarel: we are using roles not playbooks | 15:51 |
| sshnaidm | rlandy, one of problems I had is requirement for sudo when playbook tries to install packages | 15:51 |
| sshnaidm | rlandy, it means we can't run it on hosts with sudo with password | 15:51 |
| quiquell | rlandy: also we have to make notes abut gerrit ssh keys and the issue with paramiko bug | 15:51 |
| sshnaidm | rlandy, maybe we can extract all installation stuff, or to have option to skip it - for future | 15:52 |
| rlandy | sshnaidm: quiquell; sure - pls comment on the review and I will fix | 15:52 |
| quiquell | rlandy: repro review or new one ? | 15:52 |
| ykarel | quiquell, ack | 15:52 |
| rlandy | current review | 15:53 |
| rlandy | will note we can use skip-tags if running not for the first time | 15:53 |
| *** dsneddon has joined #oooq | 15:53 | |
| rlandy | sshnaidm: what are the latest images people should upload? | 15:53 |
| rlandy | for the README doc | 15:54 |
| ykarel | quiquell, ok try the revie | 15:54 |
| sshnaidm | rlandy, people shouldn't upload anything | 15:54 |
| rlandy | sshnaidm: accept? | 15:54 |
| sshnaidm | rlandy, we currently share images from openstack-nodepool tenant and they should accept | 15:54 |
| sshnaidm | rlandy, but need to request this | 15:54 |
| rlandy | sshnaidm: what should I say in the user doc? | 15:54 |
| weshay | quiquell, you there? | 15:54 |
| weshay | quiquell, did you fill out the form? | 15:55 |
| rlandy | ping on #oooq yo request images? | 15:55 |
| sshnaidm | rlandy, "You need to request sharing nodepool images from CI team providing your tenant ID and then accept the shared image" | 15:55 |
| rlandy | fine | 15:55 |
| quiquell | weshay: Haven't fill in the PTG from yet | 15:55 |
| quiquell | s/from/form/ | 15:55 |
| sshnaidm | rlandy, ah, wait.. quiquell said we have them public now | 15:55 |
| quiquell | sshnaidm: they don't have to do nothng | 15:56 |
| sshnaidm | rlandy, ok, so they should have it | 15:56 |
| rlandy | great - will just tell them to check they have access to publicly shared nodepool images | 15:57 |
| *** dsneddon has quit IRC | 16:01 | |
| *** jfrancoa has quit IRC | 16:09 | |
| chandankumar | arxcruz|ruck: kopecmartin we are back in business os_tempest gates are back to normal :-) | 16:15 |
| *** quiquell is now known as quiquell|off | 16:16 | |
| arxcruz|ruck | \o/ | 16:16 |
| *** dsneddon has joined #oooq | 16:26 | |
| weshay | ok.. CONTAINERS WIN! https://github.com/DomiStyle/docker-idrac6 | 16:27 |
| weshay | rlandy, ^ | 16:27 |
| chandankumar | weshay: http://paste.openstack.org/show/744177/ | 16:27 |
| chandankumar | weshay: this might be interesting for you | 16:27 |
| chandankumar | on collect-logs | 16:27 |
| rlandy | wow | 16:27 |
| rlandy | nice | 16:27 |
| weshay | rlandy, the docker run command works | 16:27 |
| weshay | compose does not | 16:27 |
| *** dsneddon has quit IRC | 16:32 | |
| weshay | chandankumar, nice.. sshnaidm++ | 16:33 |
| hubbot1 | weshay: sshnaidm's karma is now 13 | 16:33 |
| weshay | re: http://paste.openstack.org/show/744177/ | 16:33 |
| chandankumar | weshay: sharing is caring :-) | 16:34 |
| weshay | lolz | 16:34 |
| chandankumar | too much, /me out | 16:35 |
| *** chandankumar is now known as chkumar|out | 16:35 | |
| sshnaidm | really nice | 16:35 |
| *** agopi is now known as agopi|lunch | 16:41 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 16:47 |
| *** vinaykns has joined #oooq | 16:52 | |
| *** hamzy has quit IRC | 16:56 | |
| *** hamzy has joined #oooq | 17:00 | |
| weshay | rlandy, testing python2 now | 17:00 |
| *** vinaykns has quit IRC | 17:01 | |
| *** vinaykns has joined #oooq | 17:03 | |
| rlandy | weshay: https://review.openstack.org/#/c/631067/33/roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-based-quickstart.html.j2 | 17:04 |
| rlandy | ^^ it's a start | 17:04 |
| rlandy | weshay: my python2 failed down the line ... | 17:06 |
| rlandy | fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to load docker-compose. Try `pip install docker-compose`. Error: No module named jsonschema"} | 17:06 |
| *** dsneddon has joined #oooq | 17:06 | |
| rlandy | going back to that | 17:06 |
| *** dsneddon has quit IRC | 17:10 | |
| *** hamzy has quit IRC | 17:11 | |
| *** bogdando has quit IRC | 17:15 | |
| *** ccamacho has quit IRC | 17:18 | |
| *** kopecmartin is now known as kopecmartin|off | 17:24 | |
| *** trown is now known as trown|lunch | 17:26 | |
| *** dsneddon has joined #oooq | 17:39 | |
| *** ykarel is now known as ykarel|away | 17:41 | |
| *** agopi|lunch is now known as agopi | 17:46 | |
| *** dsneddon has quit IRC | 17:47 | |
| ssbarnea|rover | rlandy: please let me know what you think about my comment on https://review.openstack.org/#/c/631067/33/roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-based-quickstart.html.j2@59 | 17:53 |
| ssbarnea|rover | the recomandation of using unsecure ssh keys worries me, why not just using the output of `ssh-add -L` to identify user keys in a safe manner? do we have platforms where this does not work? | 17:55 |
| rlandy | ssbarnea|rover: not sure- you're the first to object to it :) | 17:56 |
| rlandy | we pick up this key in the playbook | 17:56 |
| rlandy | getting | 17:56 |
| rlandy | https://review.openstack.org/#/c/631067/33/roles/create-zuul-based-reproducer/templates/launcher-playbook.yaml.j2 | 17:56 |
| rlandy | will looking into it | 17:56 |
| rlandy | that location is an assumption in various places | 17:57 |
| rlandy | bit doesn't mean it can't be changed | 17:57 |
| ssbarnea|rover | rlandy: maybe I should drop the security guy hat, usually is not a very popular role, more of a PITA ;) -- but the reality is that i never keep my user key(s) decrypted on disk. | 17:58 |
| ssbarnea|rover | i am asking because i fixed a similar bug in old repro recently | 17:58 |
| *** derekh has quit IRC | 18:01 | |
| *** jtomasek has quit IRC | 18:10 | |
| *** dsneddon has joined #oooq | 18:10 | |
| *** jpena is now known as jpena|off | 18:15 | |
| *** trown|lunch is now known as trown | 18:21 | |
| rlandy | ssbarnea|rover: it | 18:24 |
| rlandy | 's a good suggestion | 18:24 |
| rlandy | will keep it for next sprint | 18:24 |
| ssbarnea|rover | rlandy: i suspect that fixing it would be very easy. i find this as a bug not a feature. more than willing to help. i could duplicate your WIP review and test my change and update it if it works correctly. | 18:26 |
| ssbarnea|rover | i already know the details around ssh key so it would be very easy for me to do it. | 18:27 |
| ssbarnea|rover | also it would mean that I will test the new repro :D | 18:27 |
| rlandy | ssbarnea|rover: sure go ahead | 18:27 |
| rlandy | just fixing the tar file | 18:27 |
| ykarel|away | ssbarnea|rover, i remeber you mentioned epel metalink error today, is the issue detected? i see it's affecting periodic jobs too | 18:44 |
| ykarel|away | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/d685266/job-output.txt.gz#_2019-01-29_18_11_52_922975 | 18:44 |
| ssbarnea|rover | ykarel|away: i only documented it on https://review.rdoproject.org/etherpad/p/ruckrover-sprint24 -- but I didn't do more about it because i did not see enough failures. | 18:45 |
| weshay | rlandy, I have some thoughts re: the reproducer script | 18:46 |
| rlandy | weshay: ok - changes needed? | 18:47 |
| weshay | rlandy, ya.. let me post one more thing.. then I'll point you | 18:47 |
| ykarel|away | ssbarnea|rover, good to have a priority bug, it's affecting promotions now, not sure how much though | 18:47 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades @ (1 more message) | 18:47 |
| ssbarnea|rover | ykarel|away: rdo kibana is down, so I cannot search for occurences, just reported it on irc. | 18:48 |
| ssbarnea|rover | to me this looks like a repo config issue, probably a changed gpg key? | 18:49 |
| ykarel|away | ssbarnea|rover, don't know seeing atleast two failures | 18:49 |
| ykarel|away | at https://review.rdoproject.org/zuul/status | 18:49 |
| weshay | rlandy, k https://review.openstack.org/#/c/631067/30/roles/create-zuul-based-reproducer/templates/reproducer-zuul-based-quickstart.sh.j2 | 18:50 |
| weshay | rlandy, I've removed the python interp as well.. we'll see how it goes | 18:51 |
| ykarel|away | ssbarnea|rover, see top 5 so seems actual issue affecting ovb:- https://review.rdoproject.org/zuul/builds?result=POST_FAILURE | 18:51 |
| weshay | sorry it takes me so long | 18:51 |
| * rlandy looks | 18:51 | |
| *** hamzy has joined #oooq | 18:52 | |
| weshay | we could move that git install to the launcher-env-setup playbook | 18:53 |
| rlandy | all the installs | 18:53 |
| weshay | well..except for ansible :) | 18:53 |
| rlandy | correct | 18:54 |
| rlandy | ok - let me update | 18:54 |
| rlandy | not sure about this ... quickstart.sh --install-deps | 18:54 |
| weshay | I think the docker group work should stay in the shell | 18:54 |
| weshay | rlandy, what's your concern? | 18:54 |
| * rlandy checks what else that installs | 18:55 | |
| rlandy | weshay: why keep docker work in shell? | 18:55 |
| weshay | rlandy, because it doesn't work w/ ansible | 18:55 |
| weshay | we already try to handle it ansible | 18:56 |
| weshay | it's not a good solution for a user | 18:56 |
| rlandy | hmmm ... so $USER | 18:56 |
| rlandy | is that the same as the gerrit user? | 18:56 |
| rlandy | rdo-gerrit user | 18:57 |
| rlandy | upstream gerrit user | 18:57 |
| rlandy | current system user? | 18:57 |
| rlandy | for the docker piece | 18:57 |
| weshay | $USER is the current user | 18:57 |
| weshay | the current user has to be in the docker group | 18:57 |
| rlandy | so could be diff from gerrit user | 18:58 |
| rlandy | which will be the ansible_user | 18:58 |
| weshay | ya | 18:58 |
| rlandy | ok - I can't really test this since I don;t have a split-personality setup | 18:58 |
| weshay | happy to be QE | 18:59 |
| rlandy | k- give me few minutes to update review | 18:59 |
| weshay | rlandy, sshnaidm HOT dam.. python2 ansible works and my containers are up and running | 19:00 |
| * ykarel|away out | 19:00 | |
| rlandy | nice for you | 19:01 |
| sshnaidm | weshay, \o/ | 19:01 |
| weshay | HOT dam | 19:01 |
| weshay | and I have a job running | 19:01 |
| weshay | rlandy, we have to remove the python3 interp line | 19:01 |
| weshay | WOOOOOOOOT | 19:01 |
| weshay | hot dam | 19:01 |
| rlandy | weshay: already done in latest patch | 19:02 |
| weshay | HOT dam | 19:02 |
| rlandy | weshay: plus there is a doc section on py2 vs p3 | 19:02 |
| ssbarnea|rover | weshay: rlandy : i suspect centos repos are messed or similar because install-deps no longer works: https://gist.github.com/ssbarnea/c495d4a34d8df03d7d8f55261e8abecf | 19:03 |
| rlandy | weshay: wrt install-deps ... you have two sep comments ... | 19:04 |
| rlandy | bash git/tripleo-quickstart/quickstart.sh --install-deps | 19:04 |
| rlandy | and install -y ansible python2-libselinux (adds python2-libselinux) | 19:04 |
| rlandy | you want both? | 19:04 |
| rlandy | just install-deps | 19:04 |
| rlandy | install-deps will install ... | 19:06 |
| rlandy | sudo-1.8.23-1.fc28.x86_64 | 19:06 |
| rlandy | Last metadata expiration check: 4:13:55 ago on Tue Jan 29 09:51:29 2019. | 19:06 |
| rlandy | Package git-core-2.17.2-2.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | Package gcc-8.2.1-6.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | Package iproute-4.18.0-1.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | Package libyaml-0.1.7-5.fc28.x86_64 is already installed, skipping. | 19:06 |
| weshay | rlandy, sshnaidm woot.. ansible returned w/ 0 failed | 19:06 |
| rlandy | Package libffi-devel-3.1-16.fc28.x86_64 is already installed, skipping. | 19:06 |
| weshay | first TIME | 19:06 |
| rlandy | Package openssl-devel-1:1.1.0i-1.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | Package redhat-rpm-config-110-1.fc28.noarch is already installed, skipping. | 19:06 |
| rlandy | Package python3-libselinux-2.8-1.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | Package python2-libselinux-2.8-1.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | Package python3-PyYAML-3.12-10.fc28.x86_64 is already installed, skipping. | 19:06 |
| rlandy | weshay: you can make a shehecheyanu | 19:06 |
| *** ykarel|away has quit IRC | 19:06 | |
| weshay | tags all? | 19:06 |
| weshay | what do we need to really run standalone | 19:07 |
| rlandy | if it's not a first time run, --skip-tags start | 19:07 |
| rlandy | you will need tags all or at least tags launch | 19:07 |
| rlandy | launch is not default | 19:07 |
| rlandy | weshay: did the job actually run? | 19:08 |
| rlandy | can you see zuul/gerrit status? | 19:08 |
| rlandy | there are fixes since patch 30 | 19:08 |
| weshay | hrm.. one container died | 19:09 |
| weshay | sec.. in 1-1 | 19:10 |
| rlandy | hmmm ... | 19:13 |
| *** dtantsur is now known as dtantsur|afk | 19:17 | |
| weshay | oh k.. tags launch | 19:19 |
| weshay | roger | 19:19 |
| rlandy | weshay: re: your docker comment | 19:30 |
| rlandy | sudo usermod -aG docker $USER | 19:30 |
| rlandy | echo "Add user immediately to the docker group." | 19:30 |
| rlandy | echo "This will exit the script, please re-execute." | 19:30 |
| rlandy | newgrp docker | 19:30 |
| weshay | aye | 19:30 |
| rlandy | if the add line exists the script, newgrp docker will never run | 19:30 |
| weshay | sorry? | 19:31 |
| rlandy | look at the order of lines and echos | 19:31 |
| weshay | if the user is in docker group it wont run | 19:31 |
| rlandy | so you are saying newgrp docker will exit the script??? | 19:32 |
| weshay | http://pastebin.test.redhat.com/704036 | 19:32 |
| weshay | rlandy, it does.. yes | 19:32 |
| weshay | it resets the shell | 19:32 |
| rlandy | ok - git the order now | 19:32 |
| weshay | this is getting awesome | 19:34 |
| weshay | oh wait.. I was running w/ tags all | 19:36 |
| weshay | hrm | 19:36 |
| weshay | 2019-01-29 19:34:15.949538 | primary | "msg": "Failure talking to yum: failure: repodata/repomd.xml from base: [Errno 256] No more mirrors to try.\nhttp://mirror.none.none.rdoproject.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - \"Could not resolve host: mirror.none.none.rdoproject.org; Unknown error\"" | 19:38 |
| weshay | rlandy, so I guess I need to override the mirror | 19:38 |
| rlandy | weshay:for libvirt - ack, see the doc | 19:38 |
| weshay | I assume we don't want the mirror for libvirt | 19:38 |
| weshay | k | 19:38 |
| rlandy | we do | 19:38 |
| rlandy | but not default | 19:38 |
| rlandy | weshay: https://review.openstack.org/#/c/631067/ updated with your comments | 19:40 |
| rlandy | pls also see doc explaining this | 19:40 |
| weshay | k.. cool | 19:40 |
| weshay | https://review.openstack.org/#/c/631067/30/roles/create-zuul-based-reproducer/README.md? | 19:40 |
| rlandy | yeah | 19:41 |
| weshay | oh.. gerrit | 19:41 |
| rlandy | it's easier if you render it | 19:41 |
| rlandy | not sure what the gerrit key issue is | 19:41 |
| rlandy | need to ask about that | 19:41 |
| rlandy | weshay: you need to get off patch 30 | 19:42 |
| rlandy | patch 34 | 19:42 |
| weshay | ya | 19:42 |
| weshay | sorry still not seeing it though | 19:42 |
| rlandy | not seeing what piece | 19:42 |
| weshay | re: the mirror | 19:42 |
| rlandy | oh ... | 19:43 |
| rlandy | https://review.openstack.org/#/c/631067/34/roles/create-zuul-based-reproducer/templates/launcher-env-setup-playbook.yaml.j2 | 19:43 |
| rlandy | sorry ... | 19:43 |
| rlandy | Set the options selected into EXTRA_PARAMS | 19:43 |
| rlandy | if [[ "$LIBVIRT" == "1" ]]; then | 19:43 |
| rlandy | EXTRA_PARAMS="$EXTRA_PARAMS -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org " | 19:43 |
| rlandy | fi | 19:43 |
| rlandy | or if you are just kicking the playbooks | 19:43 |
| rlandy | # To run the libvirt option (available for non-OVB jobs): | 19:44 |
| rlandy | # Clone https://github.com/openstack/tripleo-quickstart and run: | 19:44 |
| rlandy | # >> ansible-playbook launcher-playbook.yaml \ | 19:44 |
| rlandy | # -e nodepool_provider=libvirt \ | 19:44 |
| rlandy | # -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 19:44 |
| rlandy | # | 19:44 |
| rlandy | weshay: ^^ see those two pieces | 19:44 |
| rlandy | if you run the bash script with -l, it takes care of the mirror for you | 19:44 |
| weshay | ah.. I have an old copy :) | 19:45 |
| rlandy | again ... please let go of patch 30 | 19:45 |
| rlandy | it was a good one but we move on | 19:45 |
| weshay | hrm.. no I don't | 19:45 |
| weshay | EXTRA_PARAMS="$EXTRA_PARAMS -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org " | 19:45 |
| rlandy | yep | 19:45 |
| weshay | 2019-01-29 19:34:15.949538 | primary | "msg": "Failure talking to yum: failure: repodata/repomd.xml from base: [Errno 256] No more mirrors to try.\nhttp://mirror.none.none.rdoproject.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - \"Could not resolve host: mirror.none.none.rdoproject.org; Unknown error\"" | 19:46 |
| weshay | mirror.none.none | 19:46 |
| weshay | weirddd.. | 19:46 |
| weshay | will try again | 19:46 |
| rlandy | ok - will try that again as well | 19:46 |
| rlandy | weshay: still broken? | 20:00 |
| weshay | still running, I'm rerunning as is.. just to you know .. test a little | 20:00 |
| weshay | http://codesearch.openstack.org/?q=mirror_fqdn&i=nope&files=&repos= | 20:00 |
| weshay | it's the right var | 20:00 |
| rlandy | ansible-playbook ./launcher-playbook.yaml -vv --tags all -e nodepool_provider=libvirt -e libvirt_volume_path=/home/temp/images -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 20:01 |
| rlandy | ^^ has been working for me | 20:01 |
| rlandy | just need to check that the script passes the options correctly | 20:01 |
| rlandy | maybe -e should not have two options | 20:01 |
| rlandy | that could be a mistake | 20:02 |
| weshay | that seems fine | 20:03 |
| weshay | my | tee is not picking up the ansible-playbook invocation :( | 20:03 |
| *** holser__ is now known as holser|eod | 20:04 | |
| *** holser|eod has quit IRC | 20:04 | |
| weshay | same issue | 20:05 |
| weshay | rlandy, ^ | 20:05 |
| rlandy | can you paste ansible playbook output | 20:06 |
| weshay | + echo wes | 20:08 |
| weshay | wes | 20:08 |
| weshay | + echo -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 20:08 |
| weshay | nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 20:08 |
| weshay | weird | 20:10 |
| weshay | nodepool lost it's -e | 20:10 |
| *** agopi is now known as agopi|brb | 20:10 | |
| *** agopi|brb has quit IRC | 20:10 | |
| rlandy | + [[ rlandy != \r\l\a\n\d\y ]] | 20:12 |
| rlandy | + ansible-playbook /tmp/reproduce-tmp.ztryT/launcher-playbook.yaml -vv --tags all -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 20:12 |
| rlandy | weshay: ^^ wfm | 20:12 |
| rlandy | + EXTRA_PARAMS=' -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org ' | 20:13 |
| rlandy | weshay: ./reproducer-zuul-based-quickstart.sh -l | 20:13 |
| rlandy | with latest | 20:13 |
| weshay | hrm | 20:13 |
| weshay | rlandy, can you paste me the latest | 20:14 |
| weshay | that you are using | 20:14 |
| weshay | # Set the options selected into EXTRA_PARAMS | 20:15 |
| weshay | if [[ "$LIBVIRT" == "1" ]]; then | 20:15 |
| weshay | EXTRA_PARAMS+=' -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org ' | 20:15 |
| weshay | fi | 20:15 |
| weshay | also tried... | 20:15 |
| weshay | EXTRA_PARAMS="$EXTRA_PARAMS -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org " | 20:16 |
| rlandy | pasting | 20:17 |
| rlandy | weshay: http://pastebin.test.redhat.com/704065 | 20:17 |
| rlandy | and then ... ./reproducer-zuul-based-quickstart.sh -l | 20:18 |
| rlandy | weshay: better or not yet? | 20:25 |
| weshay | ++ ansible-playbook /tmp/reproduce-tmp.Hq564/launcher-playbook.yaml -vv --tags all --syntax-check -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 20:25 |
| weshay | so that looks good | 20:26 |
| weshay | will run w/o syntax-check now | 20:26 |
| weshay | ++ ansible-playbook /tmp/reproduce-tmp.Hq564/launcher-playbook.yaml -vv --tags all --syntax-check -e nodepool_provider=libvirt -e mirror_fqdn=mirror.mtl01.inap.openstack.org | 20:27 |
| rlandy | still has --syntax-check | 20:30 |
| weshay | ya.. I was just doing that to see the ansible output | 20:30 |
| weshay | reran w/o | 20:30 |
| weshay | rlandy, the only other usability thing I think of atm | 20:31 |
| weshay | is that it would be great to dump the nodepool node ip's to the workspace | 20:31 |
| weshay | so folks can ssh in | 20:31 |
| weshay | not a huge priority atm | 20:31 |
| rlandy | ok - adding the self extracting tar stuff now | 20:32 |
| weshay | cool | 20:32 |
| rlandy | doesn't seem like such a cost saving but anyways | 20:32 |
| rlandy | the self-extracting stuff | 20:32 |
| weshay | oh crud.. my gist did not save... | 20:34 |
| *** jtomasek has joined #oooq | 20:41 | |
| weshay | rlandy, hrm.. still hit it | 20:42 |
| weshay | weird | 20:42 |
| weshay | 2019-01-29 20:42:09.959926 | primary | "msg": "Failure talking to yum: failure: repodata/repomd.xml from base: [Errno 256] No more mirrors to try.\nhttp://mirror.none.none.rdoproject.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - \"Could not resolve host: mirror.none.none.rdoproject.org; Unknown error\"" | 20:42 |
| rlandy | looks like then that value is not passed | 20:44 |
| rlandy | why though, idk | 20:44 |
| rlandy | let's see how far I get with this | 20:44 |
| rlandy | are you reinstalling on the same nodes? | 20:45 |
| rlandy | kicking mine again | 20:45 |
| weshay | I was just rerunning | 20:46 |
| weshay | the script | 20:46 |
| weshay | so that does tear down the nodes | 20:46 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7 (1 more message) | 20:48 |
| weshay | TASK [debug] ******************************************************************************************************************************************************************************************************** | 20:49 |
| weshay | task path: /var/tmp/RECREATE/launcher-playbook.yaml:47 | 20:49 |
| weshay | ok: [localhost] => { | 20:49 |
| weshay | "mirror_fqdn": "mirror.mtl01.inap.openstack.org" | 20:49 |
| weshay | } | 20:49 |
| weshay | it's there ^ | 20:49 |
| rlandy | mine is on TASK [ansible-role-tripleo-ci-reproducer : prepare nodes] | 20:50 |
| weshay | wonder if it's not passed to the next one | 20:50 |
| weshay | playbook | 20:50 |
| rlandy | it should only be needed to prep the nodes | 20:50 |
| weshay | it's failing on a infra role | 20:50 |
| rlandy | when it is, that will set up the repos correctly | 20:50 |
| rlandy | correct | 20:50 |
| rlandy | install | 20:50 |
| rlandy | because the repos are looking at the wring url | 20:51 |
| rlandy | wrong | 20:51 |
| weshay | ? | 20:51 |
| weshay | can you blue for a sec | 20:51 |
| weshay | so I can share my screen | 20:51 |
| rlandy | yes | 20:51 |
| *** agopi|brb has joined #oooq | 20:54 | |
| *** agopi|brb is now known as agopi | 20:54 | |
| rlandy | weshay: https://github.com/openstack-infra/zuul-jobs/blob/master/roles/configure-mirrors/defaults/main.yaml | 21:16 |
| *** hamzy has quit IRC | 21:42 | |
| *** jtomasek has quit IRC | 22:01 | |
| *** trown is now known as trown|outtypewww | 22:01 | |
| *** dsneddon has quit IRC | 22:27 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7 (1 more message) | 22:48 |
| *** dsneddon has joined #oooq | 23:03 | |
| *** dsneddon has quit IRC | 23:08 | |
| -openstackstatus- NOTICE: http://zuul.openstack.org is not working. https://zuul.openstack.org does work. Please use that while we investigate. | 23:12 | |
| *** saneax has joined #oooq | 23:14 | |
| *** saneax has quit IRC | 23:20 | |
| *** saneax has joined #oooq | 23:22 | |
| *** vinaykns has quit IRC | 23:32 | |
| *** dsneddon has joined #oooq | 23:37 | |
| *** dsneddon has quit IRC | 23:45 | |
| *** dsneddon has joined #oooq | 23:46 | |
| *** dsneddon has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!