| *** hoonetorg has quit IRC | 00:10 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: DNM: Testing the LVM job https://review.opendev.org/739775 | 00:57 |
|---|---|---|
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: Enable cephfs snapshots https://review.opendev.org/736827 | 01:09 |
| *** maaritamm has joined #openstack-manila | 03:42 | |
| *** maaritamm has quit IRC | 03:47 | |
| *** maaritamm has joined #openstack-manila | 04:23 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: Enable cephfs snapshots https://review.opendev.org/736827 | 05:13 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: DNM: Testing the LVM job https://review.opendev.org/739775 | 05:21 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: Add native zuulv3 job for cephfs nfs https://review.opendev.org/726088 | 06:22 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: Add native zuulv3 job for cephfs nfs https://review.opendev.org/726088 | 06:24 |
| *** trident has quit IRC | 06:39 | |
| *** trident has joined #openstack-manila | 06:42 | |
| *** tosky has joined #openstack-manila | 07:33 | |
| *** dkobtsev has joined #openstack-manila | 07:41 | |
| *** dkobtsev has quit IRC | 07:44 | |
| *** maaritamm has quit IRC | 07:45 | |
| *** maaritamm has joined #openstack-manila | 07:53 | |
| *** tkajinam has quit IRC | 08:23 | |
| *** fzzf has joined #openstack-manila | 08:24 | |
| *** fzzf1 has joined #openstack-manila | 09:06 | |
| *** fzzf has quit IRC | 09:09 | |
| *** fzzf2 has joined #openstack-manila | 09:09 | |
| *** fzzf1 has quit IRC | 09:13 | |
| *** toabctl has quit IRC | 10:10 | |
| *** toabctl has joined #openstack-manila | 10:12 | |
| *** maaritamm has quit IRC | 11:08 | |
| *** maaritamm has joined #openstack-manila | 11:26 | |
| *** raildo has joined #openstack-manila | 11:37 | |
| *** vkmc has joined #openstack-manila | 11:55 | |
| *** fzzf1 has joined #openstack-manila | 12:16 | |
| *** tinwood is now known as tinwood-afk | 12:17 | |
| *** fzzf2 has quit IRC | 12:19 | |
| *** vhari has joined #openstack-manila | 12:21 | |
| *** hoonetorg has joined #openstack-manila | 12:25 | |
| *** fzzf2 has joined #openstack-manila | 12:47 | |
| *** fzzf1 has quit IRC | 12:50 | |
| *** lpetrut_ has joined #openstack-manila | 13:21 | |
| *** tinwood-afk is now known as tinwood | 13:45 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: Add native zuulv3 job for cephfs nfs https://review.opendev.org/726088 | 13:53 |
| *** dviroel has joined #openstack-manila | 13:57 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: Add native zuulv3 job for cephfs nfs https://review.opendev.org/726088 | 14:07 |
| *** lpetrut_ has quit IRC | 14:28 | |
| *** lseki has joined #openstack-manila | 14:36 | |
| *** fzzf2 has quit IRC | 15:18 | |
| *** dviroel has quit IRC | 16:07 | |
| *** gouthamr has quit IRC | 16:26 | |
| *** maaritamm has quit IRC | 16:48 | |
| *** dviroel has joined #openstack-manila | 16:51 | |
| *** gouthamr has joined #openstack-manila | 16:52 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: Add native zuulv3 job for cephfs nfs https://review.opendev.org/726088 | 17:06 |
| openstackgerrit | Tom Barron proposed openstack/manila master: Enforce policy checks getting share-type by name https://review.opendev.org/738941 | 17:36 |
| openstackgerrit | Daniel Tapia proposed openstack/manila master: [DNM] Disabling ``mark_qos_policy_group_for_deletion`` method https://review.opendev.org/740081 | 18:29 |
| tbarron | gouthamr: did you say that you have an example where concurrency=2 and the lvm job passes? | 18:46 |
| gouthamr | tbarron: yah, i celebrated early - https://zuul.opendev.org/t/openstack/build/dd7224d59eac45dab4396ad50be6da14 ran on vexxhost | 18:47 |
| tbarron | gouthamr: ah, so we need to re-run with concurrency=1 on rax (run till it lands on rax) and | 18:49 |
| tbarron | collect the system journal at the end of the job | 18:49 |
| *** ChanServ sets mode: +o gouthamr | 18:49 | |
| *** ChanServ sets mode: -o gouthamr | 18:50 | |
| tbarron | gouthamr: if that doesn't tell us anything we can ask the opendev guys to freeze a node for us | 18:50 |
| gouthamr | tbarron: we have the syslog, what other file would we need? | 18:52 |
| tbarron | gouthamr: it doesn't tell me anything, I was expecting journactl would. | 18:53 |
| tbarron | gouthamr: or do you see something in syslog? | 18:53 |
| gouthamr | tbarron: i see the reboot there too https://zuul.opendev.org/t/openstack/build/3eb91c71fa3041e98b081b9ef00a8419/log/controller/logs/syslog.txt#8915 | 18:54 |
| gouthamr | that's all though, | 18:55 |
| tbarron | like i said | 18:55 |
| tbarron | I don't *know* that the journal will tell us more but it's more modern and tightly integrated with systemd | 18:59 |
| *** maaritamm has joined #openstack-manila | 19:05 | |
| gouthamr | tbarron: hmmm, this is how syslog is being built: https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L20-L32 | 19:09 |
| tbarron | gouthamr: "with just the kernel and sudo messages" | 19:10 |
| gouthamr | tbarron: hold on, the step below extracts the whole journal | 19:10 |
| gouthamr | tbarron: https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L34-L48 | 19:10 |
| * gouthamr reads https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/templates/devstack.journal.README.txt.j2 | 19:11 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: DNM: Testing the LVM job https://review.opendev.org/739775 | 19:16 |
| gouthamr | while i check the journal, lets set the concurrency=1 ^ | 19:17 |
| *** gmann_ has joined #openstack-manila | 19:17 | |
| *** gmann_ is now known as gmann | 19:18 | |
| tbarron | gouthamr: thanks, it's perhaps easier to debug and could be a workaround until the real issue is solved | 19:29 |
| *** arne_wiebalck_ has joined #openstack-manila | 19:30 | |
| *** mnaser has joined #openstack-manila | 19:32 | |
| *** arne_wiebalck_ is now known as arne_wiebalck | 19:41 | |
| gouthamr | hmm, i don't understand why that happens | 19:45 |
| gouthamr | there's a marker file created to identify when to start processing logs - they are using the same marker file to grab the system journal to compile a syslog, as well as this export | 19:46 |
| gouthamr | oh wait, i'm tying in the wrong channel - context, tbarron and i are debugging the system journal, and the journal entries are from after the reboot | 19:47 |
| tbarron | gouthamr: this is the right channel :D | 19:47 |
| gouthamr | tbarron: yeah, but i took it elsewhere unwisely | 19:48 |
| tbarron | gouthamr: right, same marker file and syslog has the context but not the detail and journalctl is missing the full context | 19:48 |
| gouthamr | tbarron: anywho, i wonder if --no-pager makes a difference there | 19:48 |
| gouthamr | tbarron: the current execution is on inap-mtl01, not a rax - so i bet this thing will pass | 19:51 |
| tbarron | yup, bad luck to have good luck now | 19:51 |
| *** kopecmartin is now known as kopecmartin|pto | 19:52 | |
| *** dviroel has quit IRC | 19:56 | |
| *** dviroel_ has joined #openstack-manila | 19:58 | |
| gouthamr | Host and kernel version on a failed run: Linux ubuntu-bionic-rax-iad-0017881002 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | 20:00 |
| gouthamr | 16Linux ubuntu-bionic-vexxhost-ca-ymq-1-0017869042 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | 20:00 |
| gouthamr | 16 | 20:00 |
| gouthamr | ^ second line's a run which passed | 20:00 |
| gouthamr | lvm2 package versions match as well, 2.02.176-4.1ubuntu3.18.04.3 | 20:02 |
| tbarron | vexxhost has 80GB root partition; rax has 40GB, Check zuul-info/host-info.controller.yaml | 20:06 |
| tbarron | maybe, my eyes are blurry | 20:06 |
| *** carloss has joined #openstack-manila | 20:06 | |
| tbarron | but my blurry mind is thinking about the BACKING_FILE at 32000M | 20:07 |
| tbarron | DATA_DIR is probably somewhere on the root partition | 20:08 |
| *** xyang has joined #openstack-manila | 20:09 | |
| gouthamr | probably is.. | 20:11 |
| * gouthamr brb | 20:12 | |
| *** lseki has quit IRC | 20:13 | |
| *** walshh_ has joined #openstack-manila | 20:14 | |
| *** lseki has joined #openstack-manila | 20:15 | |
| *** patrickeast has joined #openstack-manila | 20:17 | |
| *** vkmc_ has joined #openstack-manila | 20:17 | |
| *** amito has joined #openstack-manila | 20:17 | |
| *** maaritamm has quit IRC | 20:17 | |
| * gouthamr is back | 20:27 | |
| gouthamr | hmmm, looks like that's been the case for a while, tbarron: https://zuul.opendev.org/t/openstack/build/d4f6e4fc780f4fefa09f46acbad8c330/log/controller/logs/df.txt - comes from a ubuntu-bionic-rax-iad node that ran a passing manila-tempest-plugin job on 2020-06-13 | 20:39 |
| gouthamr | similarly, here: https://zuul.opendev.org/t/openstack/build/6323250bae8c41ccaecb49ddc71e2021/log/controller/logs/df.txt - a ubuntu-bionic-rax-dfw node on 2020-06-12 | 20:39 |
| tbarron | gouthamr: yeah, so it's a difference and we *could* be running out of space on root on rax so it reboots but | 20:42 |
| *** maaritamm has joined #openstack-manila | 20:42 | |
| tbarron | that's just spectulative at this point and it wouldn't explain what has put us over the threshold now and not earlier | 20:42 |
| tbarron | speculative | 20:43 |
| gouthamr | yeah... we can try bumping that down in a different patch | 20:46 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: DNM: Debug LVM job's share backing file usage https://review.opendev.org/740109 | 20:48 |
| tbarron | cool | 20:50 |
| *** mnaser is now known as mnaser|ic | 20:52 | |
| gouthamr | :/ ovh picked it up ^; but our original debug patch is now on rax-iad as zuul makes a second attempt - it looks like the first attempt timed out | 20:59 |
| *** raildo has quit IRC | 21:01 | |
| * gouthamr brb | 21:06 | |
| *** mnaser|ic has quit IRC | 21:08 | |
| *** mnaser|ic has joined #openstack-manila | 21:08 | |
| *** mnaser|ic has quit IRC | 21:08 | |
| *** mnaser|ic has joined #openstack-manila | 21:08 | |
| *** mnaser|ic is now known as vexxhost | 21:08 | |
| *** gouthamr has quit IRC | 21:09 | |
| *** maaritamm has quit IRC | 21:09 | |
| *** gouthamr has joined #openstack-manila | 21:11 | |
| *** vexxhost is now known as mnaser | 21:14 | |
| *** mnaser is now known as mnaser|ic | 21:14 | |
| *** gouthamr has left #openstack-manila | 21:15 | |
| *** gouthamr has joined #openstack-manila | 21:15 | |
| *** gouthamr_ has joined #openstack-manila | 21:15 | |
| tbarron | gouthamr: gotta run but will look later. Thanks for all your work on gate! (among other things) | 21:17 |
| gouthamr | tbarron: sure thing - goodnight! | 21:17 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/manila-tempest-plugin master: Add native zuulv3 job for cephfs nfs https://review.opendev.org/726088 | 21:47 |
| *** tosky has quit IRC | 22:30 | |
| *** tkajinam has joined #openstack-manila | 22:58 | |
| *** mnaser|ic has quit IRC | 23:38 | |
| *** mnaser|ic has joined #openstack-manila | 23:38 | |
| *** mnaser|ic has quit IRC | 23:38 | |
| *** mnaser|ic has joined #openstack-manila | 23:38 | |
| *** mnaser|ic is now known as mnaser | 23:38 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!