Tuesday, 2019-12-10

*** bfournie has joined #openstack-ironic00:00
*** ociuhandu has quit IRC00:01
rpiosoTheJulia: I issued the command "openstack server create --image overcloud-full --flavor baremetal --key-name default test-server" back then.00:02
TheJulia...00:02
TheJuliarpioso: okay,, I'm going to go open a bottle of red wine, and try an experiment00:03
rpiosoTheJulia: Also issued this one -- "openstack baremetal node set {id} --property capabilities=boot_option:local,cpu_vt:true"00:03
rpiosoNote the boot_option:local00:03
rpiosoTheJulia: Sounds good. Thank you!00:03
*** rcernin_ has joined #openstack-ironic00:04
*** rcernin has quit IRC00:07
*** ociuhandu has joined #openstack-ironic00:10
*** ociuhandu has quit IRC00:14
*** ociuhandu has joined #openstack-ironic00:17
TheJuliaso it is a nested partition table00:18
TheJuliahttps://www.irccloud.com/pastebin/1JzBjEta/00:19
*** rcernin_ has quit IRC00:19
TheJuliaerr hmm00:21
*** ociuhandu has quit IRC00:21
*** ociuhandu has joined #openstack-ironic00:23
TheJuliaso not quite, the tools are interpretting it that way00:23
*** gyee has quit IRC00:24
TheJuliarpioso: so overcloud-full is a partition image, I bet you had it uploaded as a partition image and ironic did the needful to make it bootable. Would you give that a try please?00:24
rpiosoTheJulia: We're thinking alike. I've been prepping my devstack environ for another go, and it's ready.00:25
TheJuliaand maybe to try a whole disk image, use diskimage-builder to make a uefi bootable image00:25
rpiosols00:25
TheJuliaantonym was doing similar earlier it seemed00:25
* rpioso shakes his head when he enters shell commands in IRC00:25
*** rcernin has joined #openstack-ironic00:27
*** ociuhandu has quit IRC00:28
*** Goneri has joined #openstack-ironic00:29
TheJuliait happens00:30
rpiosoTheJulia: It's stackin'00:31
*** ociuhandu has joined #openstack-ironic00:31
* rpioso noticed a new project has been proposed -- go-redfish00:33
*** sshnaidm is now known as sshnaidm|afk00:35
*** ociuhandu has quit IRC00:36
*** ociuhandu has joined #openstack-ironic00:41
*** ociuhandu has quit IRC00:46
rpiosoTheJulia: The stack failed -- "Do not know what to do with overcloud-full.tar".00:51
*** ociuhandu has joined #openstack-ironic00:51
rpiosoTheJulia: Yikes! I'll fix it!00:53
*** cdearborn has quit IRC00:54
*** Goneri has quit IRC00:55
*** ricolin has joined #openstack-ironic00:56
*** ociuhandu has quit IRC00:57
openstackgerritMerged openstack/ironic-python-agent-builder master: Revert "Update tinycore from 9.x to 10.x"  https://review.opendev.org/69809801:01
*** ociuhandu has joined #openstack-ironic01:02
TheJuliasecond ci fix rechecked01:03
rpiosoTheJulia: I gzipped the beast in /opt/stack/devstack/files and updated IMAGES_URL. It's stackin' again.01:04
*** ociuhandu has quit IRC01:06
*** goldyfruit_ has joined #openstack-ironic01:09
TheJuliahave fun01:10
* TheJulia hopes CI will be good before she falls asleep01:11
*** ociuhandu has joined #openstack-ironic01:12
*** ociuhandu has quit IRC01:16
rpiosoTheJulia: Something must be wrong with IMAGES_URLS.01:19
*** ociuhandu has joined #openstack-ironic01:21
rpioso+ functions:upload_image:393               :   openstack --os-cloud=devstack-admin --os-region-name=RegionOne image create overcloud-full --property hw_rng_model=virtio --public --container-format ami --disk-format ami01:22
rpioso /opt/stack/devstack/functions: line 393: : No such file or directory01:22
*** ociuhandu has quit IRC01:26
*** ociuhandu has joined #openstack-ironic01:29
rpiosoTheJulia: In what form should I stage that tarball for stack.sh's consumption, and what are the related settings in local.conf?01:32
*** ociuhandu has quit IRC01:37
*** ociuhandu has joined #openstack-ironic01:42
*** goldyfruit_ has quit IRC01:43
*** ociuhandu has quit IRC01:47
*** ociuhandu has joined #openstack-ironic01:49
*** rloo has quit IRC01:53
*** ociuhandu has quit IRC01:57
*** ociuhandu has joined #openstack-ironic02:08
*** ociuhandu has quit IRC02:13
*** ociuhandu has joined #openstack-ironic02:14
*** bobmel has joined #openstack-ironic02:20
*** ociuhandu has quit IRC02:22
*** igordc has quit IRC02:24
*** bobmel has quit IRC02:25
*** ociuhandu has joined #openstack-ironic02:38
*** dmellado has quit IRC02:45
*** vesper11 has quit IRC02:45
*** tridde has joined #openstack-ironic02:45
*** trident has quit IRC02:46
*** vesper11 has joined #openstack-ironic02:46
*** ociuhandu has quit IRC02:47
*** dmellado has joined #openstack-ironic02:48
rpiosoTheJulia: I made some progress. I extracted the contents of the overcloud-full tarball, renamed a couple files to match devstack's image naming convention -- "-vmlinuz" and "-initrd", with a dash -- converted the qcow2 to raw .img, and created a gzipped tarball.03:07
rpiosoTheJulia: It now "Installing the bootloader for node 761dd238-3fb3-4168-bde5-a12e69cd891b on partition a192515b-f5e5-4763-ae76-4bf20db82743, EFI system partition ..."03:08
rpiosoTheJulia: However, that fails -- "ironic.common.exception.InstanceDeployFailure: Failed to install a bootloader when deploying node 761dd238-3fb3-4168-bde5-a12e69cd891b. Error: {'message': "Command execution failed: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command.\nCommand: mount /dev/sda2 /tmp/tmpHwiv0H\nExit code: 32\nStdout: u''\nStderr: u'mount: wrong fs03:11
rpiosotype, bad option, bad superblock on /dev/sda2,\\n       missing codepage or helper program, or other error\\n\\n"03:11
rpiosoTheJulia: Do you have a suggestion on how to address that one?03:11
*** michchap has joined #openstack-ironic03:16
*** ociuhandu has joined #openstack-ironic03:47
*** ociuhandu has quit IRC03:51
*** rh-jelabarre has quit IRC04:08
*** mkrai has joined #openstack-ironic04:23
*** ociuhandu has joined #openstack-ironic04:41
*** ociuhandu has quit IRC04:46
openstackgerritSteve Baker proposed openstack/metalsmith master: Implement a instance_config.NovajoinConfig  https://review.opendev.org/69699304:49
*** tzumainn has quit IRC04:53
*** ociuhandu has joined #openstack-ironic05:35
*** ociuhandu has quit IRC05:40
kaifengI have created a story for the ipxe discussion: https://storyboard.openstack.org/#!/story/2007003 and put some context in.06:13
kaifengTheJulia dtantsur|afk ^^06:14
kaifengjust let me know if you have any concern.06:15
*** bobmel has joined #openstack-ironic06:22
openstackgerritKaifeng Wang proposed openstack/ironic master: Explicitly use ipxe as boot interface for iPXE testing  https://review.opendev.org/69814606:25
*** bobmel has quit IRC06:26
*** ociuhandu has joined #openstack-ironic06:29
*** jtomasek has joined #openstack-ironic06:31
*** ociuhandu has quit IRC06:34
openstackgerritKaifeng Wang proposed openstack/ironic master: Fix ipxe interface to perform ipxe boot without ipxe_enabled enabled  https://review.opendev.org/69763406:55
*** ociuhandu has joined #openstack-ironic07:02
*** ociuhandu has quit IRC07:06
arne_wiebalckGood morning kaifeng and ironic!07:26
*** ociuhandu has joined #openstack-ironic07:26
*** ociuhandu has quit IRC07:32
*** bobmel has joined #openstack-ironic07:33
*** mkrai has quit IRC07:35
kaifenggood morning, arne_wiebalck o/07:36
*** mkrai has joined #openstack-ironic07:38
*** pcaruana has joined #openstack-ironic07:39
*** ociuhandu has joined #openstack-ironic07:46
*** rcernin has quit IRC07:54
mkraigood morning ironic arne_wiebalck o/08:23
mkraikaifeng, gm08:23
*** tesseract has joined #openstack-ironic08:27
*** khansa has joined #openstack-ironic08:28
kaifenggood morning mkrai o/08:35
arne_wiebalckhey mkrai o/08:37
*** gudrutis2 has joined #openstack-ironic08:38
*** rpittau|afk is now known as rpittau08:41
rpittaugood morning ironic! o/08:41
kaifenghey rpittau o/08:42
rpittauhey kaifeng :)08:42
kaifengrpittau: do you know what does if [ -h $something ] mean?08:43
rpittaukaifeng: that should check if the file is a symlink08:44
rpittau$something ^08:44
kaifengrpittau: ack, thanks!08:44
*** mkrai has quit IRC08:45
*** mkrai has joined #openstack-ironic08:46
*** iurygregory has joined #openstack-ironic08:46
iurygregorygood morning Ironic o/08:46
kaifenghey iurygregory o/08:47
iurygregorykaifeng, o/08:47
*** ociuhandu has quit IRC08:48
kaifengit seems ironic-tempest-ipa-{wholedisk,partition}-bios-agent_ipmitool-tinyipa* are broken due to some ssh authentication issue in paramiko.08:52
iurygregorykaifeng, what patch? there were some problems in ipa yesterday now things may be better...08:52
iurygregoryJulia pushed https://review.opendev.org/#/c/698098/08:53
patchbotpatch 698098 - ironic-python-agent-builder - Revert "Update tinycore from 9.x to 10.x" (MERGED) - 1 patch set08:53
rpittauhey iurygregory :)08:54
iurygregoryrpittau, o/08:54
kaifengiurygregory: i have seen this in my patches and it seems there in the last week.08:54
mkraigood morning iurygregory08:54
mkraigood morning rpittau08:54
iurygregorykaifeng, ouch =(08:55
kaifengiurygregory: doesn't look like related to my change..08:55
iurygregorymkrai, o/08:55
mkraiI am trying to build an IPA image but it fails with "/tmp/in_target.d/install.d/60-ironic-agent-install: line 13: /usr/share/ironic-python-agent/imagebuild/common/generate_upper_constraints.sh: No such file or directory"08:55
iurygregorykaifeng, do you have the links?08:55
kaifengthe 69814608:56
mkraiDo I need to install anything before building the image?08:56
kaifengmkrai: maybe you need a new version? I have seen such report recently.08:57
mkraikaifeng, new version of what?08:57
mkraikaifeng, my setup is running stein08:57
kaifengmkrai: https://storyboard.openstack.org/#!/200679008:58
*** sshnaidm|afk is now known as sshnaidm08:58
iurygregorykaifeng, in 698146 i don't see any logs to look .-.08:58
iurygregorynvm08:59
iurygregoryforgot I can use zuul XD08:59
kaifengiurygregory: oh, its the in the top of zuul.openstack.org :)08:59
iurygregoryyeah =)08:59
*** priteau has joined #openstack-ironic09:00
kaifengmkrai: i have successfully built dib image with the latest ipa-b a few weeks before, just couldn't remember which version i was using09:01
mkraikaifeng, /usr/share/ironic-python-agent/ i don;t have this folder09:03
mkraikaifeng, I installed ipa from source09:03
iurygregorykaifeng, maybe this was the problem with the rescue tests that dtantsur|afk was talking about09:04
iurygregorynot sure if the fix is in place...09:04
kaifengiurygregory: is it https://review.opendev.org/698011 ?09:06
patchbotpatch 698011 - ironic-tempest-plugin - Actually test rescue in the standalone job - 1 patch set09:06
kaifengthat sounds like we didn't really test rescue in standalone before09:06
iurygregoryyeah...09:06
kaifengif we have issue in the rescue test, then this patch would beat ci harder :)09:07
kaifengmkrai: have you tried pip3 install --user ironic-python-agent-builder and goes from there?09:08
mkraikaifeng, no, i used dib directly09:11
kaifengmkrai: i think you need to use a lower dib version since ironic-agent has been removed in the train cycle.09:12
kaifengand with an older ipa source09:13
mkraikaifeng, ok thanks i will try that09:14
*** lucasagomes has joined #openstack-ironic09:15
*** ociuhandu has joined #openstack-ironic09:17
openstackgerritPradip Kadam proposed openstack/ironic master: DRAC : idrac-redfish inspect updates pxe port  https://review.opendev.org/69332809:20
*** ociuhandu has quit IRC09:22
openstackgerritRachit Kapadia proposed openstack/ironic master: DRAC: Drives conversion from raid to jbod  https://review.opendev.org/67103809:24
*** dougsz has joined #openstack-ironic09:30
kaifengiurygregory: i suspect you are right, the broken jobs seem related with rescue tests..09:31
openstackgerritMerged openstack/ironic-python-agent-builder master: Update tinycore mirrors  https://review.opendev.org/69804709:31
iurygregorythis would be good XD since at least we know09:31
kaifengfeel like tinycore has some issue on providing ssh service after bumping09:31
kaifengiurygregory: agreed :)09:33
*** alexmcleod has joined #openstack-ironic09:34
rpittaukaifeng: after moving back to 9.x (merged already) it should be ok, I see all the jobs that were failing are passing now09:36
*** derekh has joined #openstack-ironic09:37
*** ociuhandu has joined #openstack-ironic09:42
openstackgerritRachit Kapadia proposed openstack/ironic master: DRAC: Drives conversion from raid to jbod  https://review.opendev.org/67103810:02
*** priteau has quit IRC10:05
*** dtantsur|afk is now known as dtantsur10:15
*** derekh has quit IRC10:20
dtantsurmorning ironic10:26
*** etingof has quit IRC10:26
openstackgerritDmitry Tantsur proposed openstack/ironic master: CI: disable IPv6 in neutron  https://review.opendev.org/69801210:26
iurygregorymorning dtantsur10:26
dtantsurarne_wiebalck, rpittau, kaifeng could you check ^^ asap10:26
openstackgerritDmitry Tantsur proposed openstack/ironic master: CI: re-enable testing rescue  https://review.opendev.org/69820710:27
rpittaudtantsur: done, I skipped that, so much chaos today10:27
*** etingof has joined #openstack-ironic10:29
arne_wiebalckdtantsur: done10:30
openstackgerritDmitry Tantsur proposed openstack/ironic master: CI: re-enable testing rescue  https://review.opendev.org/69820710:32
kaifenguh, late for the party, i just can resist food :)10:43
rpittau:D10:44
dtantsurno need to resist food!10:44
iurygregoryfood is always good =D10:45
kaifengespecially in a state of hungry :)10:46
kaifengrpittau: the revert patch (698098) merged before my propose a patch, but I still see the jobs failing in 69814610:51
dtantsurI think we have a lot of overlapping problems10:52
dtantsurRescue hard-broken is only one of them10:52
rpittaukaifeng: that's a different issue, yeah :/10:52
* dtantsur is impressed that some patches actually do merge10:53
iurygregorydtantsur, it's magic!10:54
*** dsneddon has quit IRC11:08
openstackgerritKaifeng Wang proposed openstack/ironic master: Remove the [pxe]ipxe_enabled configuration option  https://review.opendev.org/69681011:11
*** ociuhandu has quit IRC11:13
*** ociuhandu has joined #openstack-ironic11:13
openstackgerritKaifeng Wang proposed openstack/ironic master: Start removing ipxe support from the pxe interface  https://review.opendev.org/69666611:13
*** dsneddon has joined #openstack-ironic11:17
mkraikaifeng, I used ironic-python-agent-builder still the same issue exist. http://paste.openstack.org/show/787361/11:19
dtantsurmkrai: what's the command line you're using?11:19
mkraidtantsur,  ironic-python-agent-builder -o my-ipa  -e devuser debian11:19
dtantsurmkrai: which version of IPA-builder do you have? make sure you have the latest11:20
*** bobmel has quit IRC11:20
mkraidtantsur, 1.0.011:21
*** bobmel has joined #openstack-ironic11:21
mkraidtantsur, 1.10 is the latest? https://tarballs.openstack.org/ironic-python-agent-builder/11:21
mkrai* 1.1.011:22
dtantsurmkrai: grab 1.1.0, I think it'll fix your issue11:22
mkraiOk thanks dtantsur11:22
*** dsneddon has quit IRC11:23
kaifengchecked my building env, i was using ipa-b 1.1.011:24
*** ociuhandu has quit IRC11:28
mkraidtantsur, thanks it worked11:32
dtantsurI'm glad to hear it11:33
kaifengdtantsur: do you think it's a duplicate thing if we put some useful variables and elements in the ipa-b documents?11:36
dtantsurkaifeng: you mean, duplicating with DIB?11:36
dtantsurI think it's fine as long as they're really relevant to our case11:37
dtantsurlikely to be used11:37
kaifengdtantsur: i am identifying an issue that the network name can't be renamed to the name of consistent naming11:37
kaifengi am seeing ipa collected eth0 instead of enN or ensN or such thing, and at last identified there is an element stable-interface-names helps with this11:39
kaifengdtantsur: i don't want to duplicate docs with DIB, but if we have some pointers there, that may help users building their images. before today, i even don't know this element exists.11:40
dtantsurme neither11:40
dtantsuryeah, this one seems potentially relevant11:40
*** ijw has joined #openstack-ironic11:43
kaifengdtantsur: okay, i may collect something for a while and then put an update to the ipa-b docs.11:43
*** ijw has quit IRC11:47
dtantsur++ thx11:52
* dtantsur IKEA time, back later11:53
*** dtantsur is now known as dtantsur|ikea11:54
mkraikaifeng, I created the custom ipa image with devsuser, set it to cleaning and it is stuck in clean-wait11:54
mkraikaifeng, How can I check if IPA is running fine or not?11:54
mkraikaifeng, this document https://docs.openstack.org/ironic-python-agent/latest/admin/troubleshooting.html#diskimage-builder-dib only tells how to create images for debugging purpose11:55
kaifengmkrai: you are using real machine right?11:55
mkraikaifeng, yes11:56
mkraiI can see the console11:56
mkraiIt prompts "debian login"11:56
kaifengmkrai: usually i use kvm to see what happens if something goes wrong11:56
mkraikaifeng, I saw PXE and tftp happened fine11:56
kaifengyou can use the user to login into the ramdisk11:57
mkraikaifeng, it prompts for password :(11:57
kaifengthe value you set to DIB_DEV_USER_USERNAME/PASSWORD before image building11:57
mkraikaifeng, Ohh I didn't set password11:57
kaifengoh sure, you'll need that, password or publickey11:58
*** dsneddon has joined #openstack-ironic11:58
kaifengyou can refer to the dib documentation for the environmental variable required11:58
kaifengshould be under the devuser element11:58
mkraikaifeng, ok i will check that11:59
*** lucasagomes is now known as lucas-hungry12:02
kaifengmkrai: since the pxe goes well, make sure you have correct callback url to ironic api from the ipa. typically this means ipa is unable to reach ironic api to continue steps followed.12:07
mkraikaifeng, Ok I will check that once I get access to IPA12:08
mkraikaifeng, If the nodes become available it means the IPA is running just fine right?12:09
kaifengmkrai: yeah, any state not ends with "failed" implies a successful op12:11
mkraikaifeng, ok so I already have such ramdisk images which move node to available. My actual issue is the node provisioning fails with12:13
mkrai"no boot filename received"12:13
kaifengmkrai: it's the same issue yesterday?12:14
mkraikaifeng, yes :(12:15
kaifengmkrai: i suggest try localboot to make it work, netboot in neutron network needs more work.12:15
mkraikaifeng, by more work do you mean to create an OVS bridge and so12:16
mkraikaifeng, I tried localboot yesterday suggested by dtantsur|ikea, it also failed12:16
kaifengmkrai: so stuck in wait callback?12:17
kaifengmkrai: if you successfully provision a node to local boot, but still failed to startup, make sure the instance image is bootable from disk.12:19
mkraikaifeng, no it dfailed in provisioning itself12:19
mkraikaifeng, http://paste.openstack.org/show/787334/12:20
kaifengmkrai: this is ip accessing instance image from swift failed12:21
kaifengs/ip/ipa/12:22
*** ociuhandu has joined #openstack-ironic12:23
kaifengmkrai: sorry i got to go now, i suggest you use iscsi which has lesser requirements on the environment.12:23
mkraikaifeng, ok thanks for your help12:24
mkraikaifeng, bye!12:24
kaifengmkrai: if you are sticking to agent driver, make sure ipa has access to the glance api, and tempurl thing is set correctly.12:24
*** kaifeng is now known as kaifeng|afk12:24
*** ociuhandu has quit IRC12:27
*** bobmel has quit IRC12:29
*** bobmel has joined #openstack-ironic12:30
*** mkrai has quit IRC12:48
*** bfournie has quit IRC12:51
*** rh-jelabarre has joined #openstack-ironic12:55
*** lucas-hungry is now known as lucasagomes13:06
*** rloo has joined #openstack-ironic13:35
*** goldyfruit_ has joined #openstack-ironic13:37
*** yaawang has quit IRC13:39
*** yaawang has joined #openstack-ironic13:39
*** goldyfruit___ has joined #openstack-ironic13:41
*** bfournie has joined #openstack-ironic13:42
etingofTheJulia, o/ x/sushy-oem-idrac - does it need proper OS docs? I understand it should NOT have reno, but what about docs?13:42
*** Goneri has joined #openstack-ironic13:43
*** goldyfruit_ has quit IRC13:44
*** tkajinam has joined #openstack-ironic14:01
*** zaneb has quit IRC14:09
*** dsneddon has quit IRC14:12
*** dsneddon has joined #openstack-ironic14:15
*** dsneddon has quit IRC14:20
TheJuliadocs could be useful, but they would be published to rtfd14:22
*** dsneddon has joined #openstack-ironic14:22
*** mkrai has joined #openstack-ironic14:23
iurygregorygood morning TheJulia o/14:23
TheJuliado we need to drop tests and shove https://review.opendev.org/#/c/698012/ in ?14:30
patchbotpatch 698012 - ironic - CI: disable IPv6 in neutron - 2 patch sets14:30
*** dsneddon has quit IRC14:33
*** ociuhandu has joined #openstack-ironic14:35
*** mmethot has joined #openstack-ironic14:35
rpittauTheJulia: I just rechecked that, it's kind of needed to fix CI14:35
*** mmethot is now known as mmethot|conferen14:35
*** pcaruana has quit IRC14:36
TheJuliaI understand, I'm wondering if we're going to be able to merge to master at all without ipa fix merging14:37
rpittauTheJulia: it failed on standalone only14:37
*** tzumainn has joined #openstack-ironic14:39
TheJuliano, it could fail on any job that had rescue testing enabled14:40
TheJuliaof course, yes, the last attempt only failed on standalone14:40
*** tkajinam has quit IRC14:52
*** zaneb has joined #openstack-ironic14:52
TheJuliait looks like the rescue explit tests in standalone just completely tanked14:53
TheJulianever got far enough to actually test rescue :(14:54
*** igordc has joined #openstack-ironic14:58
rpittauyeah but I'm optymist for some obscure reason14:59
*** dsneddon has joined #openstack-ironic15:00
TheJuliaif it doesn't pass on the first round of trying to roll throught the gate today, I'll just force it in. We can't be dead in the water for a week15:00
rpittauwondering if it would make sense to have a recheck on single jobs in zuul, and wondering if I'm the first one to think about that15:00
rpittauTheJulia: agreed15:00
TheJulianot the first and there is a... umm... fundimental disagreement with the zuul team on that.15:00
TheJuliaWe're alsos uper job heavy15:00
rpittauoh interesting15:01
TheJuliasuper job heavy15:01
*** pcaruana has joined #openstack-ironic15:01
*** zaneb has quit IRC15:02
* rpittau loves poking hornet's nests15:02
* iurygregory likes rpittau idea15:03
* iurygregory is not going to poke any hornet's nests 15:04
*** dsneddon has quit IRC15:05
iurygregoryTheJulia, the sync command would be after the rmtree?15:05
* iurygregory is not sure where to put, I was thinking before the first unmount on fnally block or in the end of the try15:06
*** ijw has joined #openstack-ironic15:10
*** seba has joined #openstack-ironic15:14
*** bobmel has quit IRC15:15
*** bnemec has quit IRC15:15
*** munimeha1 has joined #openstack-ironic15:18
openstackgerritJulia Kreger proposed openstack/ironic master: Pre-shared agent token  https://review.opendev.org/69260915:19
openstackgerritRiccardo Pittau proposed openstack/ironic-python-agent-builder master: Upgrade syslinux version  https://review.opendev.org/69827215:23
openstackgerritEric Fried proposed openstack/ironic master: Update nova os-server-external-events response logic  https://review.opendev.org/69827315:28
iurygregoryrpittau, do you think in the case of we have the efi_system_part_uuid but for some reason we can't get the partition we should search for a valid partition? http://paste.openstack.org/show/787377/15:28
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Restrict ability to change owner on provisioned node  https://review.opendev.org/69807815:31
*** hjensas has quit IRC15:35
*** ijw has quit IRC15:40
*** dsneddon has joined #openstack-ironic15:40
TheJuliaiurygregory: eh, after unmounts15:46
rpittauiurygregory: mmm if the uuid doesn't correspond to a partition there may be something very wrong15:48
iurygregoryTheJulia, rpittau tks!15:55
rpittauiurygregory: that is taken directly from the partition creation, I doubt we will end up there if anything goes wrong during disk partitioning15:57
*** goldyfruit_ has joined #openstack-ironic15:57
*** goldyfruit___ has quit IRC16:00
*** ociuhandu has quit IRC16:02
*** dtantsur|ikea is now known as dtantsur16:02
dtantsurmorning TheJulia16:04
dtantsurso, https://review.opendev.org/#/c/698012/ doesn't seem to help, does it? >_<16:04
patchbotpatch 698012 - ironic - CI: disable IPv6 in neutron - 2 patch sets16:04
dtantsurdo we have any options left other than disabling the standalone job?16:05
dtantsurrpittau: ^^^16:05
iurygregorydtantsur, it does I think16:06
iurygregoryironic-standalone is green on zuul16:06
*** zaneb has joined #openstack-ironic16:06
dtantsuriurygregory: it's not on the last run16:06
dtantsurand again we see DHCP issues https://75e2faadb9d5832df0cd-a7818f858a3bcdda33f05669617cb625.ssl.cf2.rackcdn.com/698012/2/check/ironic-standalone/172cef9/controller/logs/ironic-bm-logs/node-5_console_log.txt.gz16:06
dtantsurjust this time there are no IPv6 addresses, so no addresses at all16:07
iurygregoryD:16:07
*** ijw has joined #openstack-ironic16:09
dtantsurwhat's even more suspicious: PXE boot retries (that we've recently introduced) don't help16:09
dtantsurso we have 3 failures to DHCP in a row16:09
dtantsursee e.g. node-1 logs in https://f67ae6e1c6d7074fe459-ba5467d2de73878d3dd4296de3d69c75.ssl.cf2.rackcdn.com/698012/1/check/ironic-standalone/142504c/controller/logs/ironic-bm-logs/16:09
rpittaushould we force ipv4 ?16:11
*** igordc has quit IRC16:11
openstackgerritDmitry Tantsur proposed openstack/ironic master: CI: do not enable rescue on indirect jobs  https://review.opendev.org/69820716:12
dtantsurrpittau: we tried in https://review.opendev.org/#/c/698012/16:12
patchbotpatch 698012 - ironic - CI: disable IPv6 in neutron (ABANDONED) - 2 patch sets16:12
dtantsurwe end up with no addresses at all16:12
rpittaudtantsur: sorry, yeah, still too much enthropy today16:12
openstackgerritJulia Kreger proposed openstack/ironic master: WIP ipv6 jobs  https://review.opendev.org/61316916:14
*** zaneb has quit IRC16:15
*** bobmel has joined #openstack-ironic16:16
TheJuliahmmmmm16:16
dtantsursomething that surprises me is that according to http://zuul.openstack.org/builds?job_name=ironic-standalone the job does pass most of the time16:16
dtantsurwhile my feeling is that it fails in 90% cases16:17
dtantsurhttp://zuul.openstack.org/builds?job_name=ironic-standalone-ipa-src shows a worse picture, but still not catastrophic16:17
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: Avoid grub2-install when on UEFI boot mode  https://review.opendev.org/69691416:18
TheJuliaare we sure that neutron didn't bind ports or update dhcp like 10-15 seconds after we booted the VM?16:18
*** iurygregory has quit IRC16:18
TheJulialike 1000% sure?16:18
rpittauTheJulia: that's what I was checking too16:18
TheJuliabecause we've seen it enough that its a known thing that can happen, and why we wanted to callback stuffs16:18
*** iurygregory has joined #openstack-ironic16:19
*** ricolin has quit IRC16:19
dtantsurTheJulia: I'm not sure at all16:20
dtantsurnow, what surprises me as well is http://zuul.openstack.org/builds?job_name=ironic-standalone&change=69301716:20
dtantsurthis is etingof's CI job patch. it shows a surprising failure rate..16:20
dtantsurTheJulia: the neutron hypothesis would explain why the failures happen more often on our most busy job16:21
dtantsurand it's relatively easy to test it!16:21
TheJulianasty neutron errors16:22
TheJulialike... around dhcp updates16:22
etingofthat's perfectly understandable - ipmi job is trying to keep its foot in the door16:22
rpittauI'm seeing an exception in the neutron dhcp agent logs, not sure it's related16:23
openstackgerritDmitry Tantsur proposed openstack/ironic master: DNM raise port_setup_delay substantially  https://review.opendev.org/69828916:23
dtantsuretingof: that grumpy old ipmi!16:23
dtantsurTheJulia: let's test your idea ^^^16:23
TheJulianot going to always work, fun error incoming16:23
rpittauDec 10 11:09:02.674224 ubuntu-bionic-rax-ord-0013309750 neutron-dhcp-agent[20060]: ERROR neutron.agent.dhcp.agent [-] Unable to reload_allocations dhcp for 2d09d457-e7ab-490c-92a6-55518600d486.: multiprocessing.managers.RemoteError:16:23
TheJuliahttps://www.irccloud.com/pastebin/LUhfa1KS/16:23
dtantsurfun? I like fun16:23
* TheJulia hands dtantsur whiskey16:23
TheJuliaand points at link16:24
rpittauthere you go :)16:24
dtantsuruuuuuuuuuuuugh16:24
dtantsurwho is going to #openstack-neutron? everyone? :)16:24
rpittau:D16:24
iurygregorylets join the party yay16:24
iurygregorygrab the drink that you like XD16:25
* dtantsur wants a shot of sweet nice working CI16:25
iurygregory^ this is the gift for xmas...16:26
rpiosoGood morning, ironic16:26
rpittauhey rpioso :)16:27
iurygregorymorning rpioso o/16:27
*** hjensas has joined #openstack-ironic16:29
rpiosodtantsur: I made more progress getting a devstack local install. I used the RDO overcloud-full tarball and modified it to make stack.sh at least somewhat happy with it. The deployment failed toward the very end -- "ironic.common.exception.InstanceDeployFailure: Failed to install a bootloader when deploying node 761dd238-3fb3-4168-bde5-a12e69cd891b. Error: {'message': "Command execution failed: Installing GRUB216:29
rpiosoboot loader to device /dev/sda failed with Unexpected error while running command.\nCommand: mount /dev/sda2 /tmp/tmpHwiv0H\nExit code: 32\nStdout: u''\nStderr: u'mount: wrong fs type, bad option, bad superblock on /dev/sda2,\\n       missing codepage or helper program, or other error\\n\\n"16:29
rpiosodtantsur: Any idea how to address that one?16:29
rpiosoiurygregory, rpittau: \o16:29
dtantsurrpioso: overcloud-full is a partition image, did you add kernel/ramdisk to glance?16:29
*** zaneb has joined #openstack-ironic16:30
rpiosodtantsur: Yes, I believe they're there. Three elements were uploaded.16:30
dtantsurrpioso: and linked from the main image?16:31
rpiosodtantsur: How can I tell?16:31
rpiosodtantsur: The main image has a ramdisk_id property which contains the ID of the ramdisk image.16:32
dtantsurrpioso: compare http://paste.openstack.org/show/787384/16:33
rpiosodtanstur: Same with the kernel_id.16:33
dtantsurah, good16:33
dtantsurworth double-checking that the right image was uploaded as the root one16:35
rpiosodtantsur: I don't have a cirros-0.4.0-x86_64-disk like thing, though. Just three items from overcloud-full are uploaded.16:35
dtantsurrpioso: yeah, I should you it for an example, I don't have an environment with overcloud-full at hand16:36
*** gyee has joined #openstack-ironic16:37
rpiosodtantsur: http://paste.openstack.org/show/787385/16:39
rpiosodtantsur: What are cirros-d160722-x86_64-disk and cirros-d160722-x86_64-uec ?16:40
dtantsurrpioso: cirros images, used for testing16:40
dtantsurdisk is a whole disk image, uec - partition16:40
rpiosodtantsur: I see.16:41
*** goldyfruit___ has joined #openstack-ironic16:41
rpiosodtantsur: Do you know what /dev/sda2 is expected to contain?16:43
dtantsurrpioso: I assume it's your root partition16:43
*** goldyfruit_ has quit IRC16:44
rpiosodtantsur: Might a lack of root device hints be causing that error?16:45
rpiosoMy impression is that the root partition was successfully written to the disk before the mount. Correct?16:45
TheJuliais raw streaming turned on?16:46
dtantsurrpioso: it was, that's why I suspect the contents of the image.16:46
rpiosoTheJulia: Please elaborate. I'm using iSCSI deploy.16:47
dtantsurroot hints only define the device itself, they have nothing to do with partitions16:47
dtantsurif iSCSI, then raw streaming is off16:47
*** mmethot|conferen is now known as mmethot16:48
TheJuliaI'd second image contents, I'd try to make sure that it was the qcow that was upload and that the qcow can be read/mounted locally16:48
rpiosoOh! I have some further insight. I used qemu-img to convert the qcow2 to raw in order to placate devstack's deployment. At least I believe I was placating it.16:50
rpiosoI had to rename the kernel and ramdisk partitions and create a new zipped tarball.16:50
rpiosoIt appears functions.update_image expects a dash in the ramdisk and kernel file names. Should I just rename the qcow2 by removing its extension?16:52
dtantsurrpioso: it should be fine to just use qcow2. which problems did you have with it?16:52
rpiosodtantsur: The file names.16:52
dtantsurrpioso: I'm quite confident the file names do not affect ironic16:52
rpiosodtantsur: They affect the devstack stack.sh.16:53
dtantsurbut renaming files don't require converting between the formats, so I'm still a bit confused16:53
rpiosoI made an assumption that no extension implied a desire for raw.16:53
dtantsurrpioso: you can just upload  the image yourself, it's not very tricky. I think we have the docs somewhere.16:53
*** ijw_ has joined #openstack-ironic16:53
TheJuliahow about a diff of what you did to the devstack plugin because there is code to loop through stuff and do things, and it sounds like you might bave run afoul of that code16:53
TheJuliaand then gotten bad content uploaded or uploaded with the wrong flags16:54
dtantsurcheck the user image part in https://docs.openstack.org/ironic/latest/install/configure-glance-images.html16:54
rpiosohttps://opendev.org/openstack/devstack/src/commit/ede8b1269cb03178c31f700200309dd76b00dd5a/functions#L297-L31516:55
rpiosodtantsur: I'll follow those instructions and give it another go.16:56
rpiosodtantsur, TheJulia: Thank you!16:56
*** ijw has quit IRC16:58
*** tesseract has quit IRC16:58
*** mkrai has quit IRC17:02
*** iurygregory has quit IRC17:02
*** lucasagomes has quit IRC17:04
*** bobmel has quit IRC17:11
* rpioso grins at the penguins displayed during the successful local boot17:15
rpiosodtantsur, TheJulia: Success! \o/17:15
dtantsurgreat!17:16
rpiosoMany thanks!17:17
TheJuliawoot17:18
*** hjensas has quit IRC17:20
rpiosoDo you know the overcloud-full CentOS 7 root or other creds?17:23
rpiosoIt's actually accepting my keystrokes.17:23
TheJuliaroot and the key you supplied for the deploy?17:24
rpiosoSSH?17:24
*** ociuhandu has joined #openstack-ironic17:25
TheJuliayes17:25
*** ociuhandu has quit IRC17:29
rpittaugood night! o/17:29
*** rpittau is now known as rpittau|afk17:29
rpiosorpittau|afk: Nite :-)17:32
* TheJulia wants to go back to sleep17:35
TheJuliadaystar move faster!17:35
TheJuliadtantsur: on a plus side https://review.opendev.org/#/c/692609/ has a +1 from zuul17:37
patchbotpatch 692609 - ironic - Pre-shared agent token - 7 patch sets17:37
dtantsurat least something has a +1 from zuul17:37
* dtantsur puts on his queue17:37
TheJulia\o/17:38
dtantsurrpioso: no default creds. you need to use ssh keypairs (if using with nova)17:38
dtantsurtime to go now, g'night17:38
*** dtantsur is now known as dtantsur|afk17:38
rpiosodtantsur|afk: Thanks. No nova, though.17:38
dtantsur|afkrpioso: try metalsmith :)17:39
dtantsur|afk(disclaimer: I wrote it)17:39
dtantsur|afkit's a CLI for ironic without nova, can handle things like networks, ssh keys, etc17:39
* dtantsur|afk goes for real now17:39
rpiosodtantsur|afk: I know well :-) Is it easier to deploy on real HW than devstack?17:39
*** sshnaidm is now known as sshnaidm|afk17:50
*** khansa has quit IRC17:57
openstackgerritJulia Kreger proposed openstack/ironic master: Pre-shared agent token  https://review.opendev.org/69260917:58
*** dougsz has quit IRC17:58
openstackgerritJulia Kreger proposed openstack/ironic master: WIP: Enable agent_token for virtual media boot  https://review.opendev.org/69758417:58
openstackgerritJulia Kreger proposed openstack/ironic master: WIP: Send our token back to the agent  https://review.opendev.org/69758517:58
openstackgerritMark Goddard proposed openstack/ironic master: WIP: Decompose core deploy step  https://review.opendev.org/69830917:59
mgoddardarne_wiebalck, dtantsur|afk: ^ decomposed core deploy step, booted a node using the iscsi deploy interface :)18:00
*** bobmel has joined #openstack-ironic18:06
rajinir<TheJulia>, dtansur,rpioso and everyone , Dell Ironic CI is back online. I have updated the whiteboard18:06
*** ijw_ has quit IRC18:07
*** ijw has joined #openstack-ironic18:09
arne_wiebalckmgoddard: wow, nice!18:09
rpiosorajinir: Thank you!18:11
rpiosoiurygregory: ^^^18:11
*** tzumainn has quit IRC18:12
TheJuliarajinir: thanks! one less email to send out \o/18:20
*** goldyfruit_ has joined #openstack-ironic18:22
*** tzumainn has joined #openstack-ironic18:24
*** goldyfruit___ has quit IRC18:25
*** ijw has quit IRC18:26
*** ijw has joined #openstack-ironic18:27
*** ijw has quit IRC18:27
*** ijw has joined #openstack-ironic18:28
*** bobmel has quit IRC18:38
*** ociuhandu has joined #openstack-ironic18:39
*** bobmel has joined #openstack-ironic18:41
*** ociuhandu has quit IRC18:44
*** ijw_ has joined #openstack-ironic18:45
*** mmethot has quit IRC18:46
*** mmethot has joined #openstack-ironic18:46
*** bobmel has quit IRC18:46
*** ijw has quit IRC18:48
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Restrict ability to change owner on provisioned node  https://review.opendev.org/69807818:56
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Add owner to allocations and create relevant policies  https://review.opendev.org/69670719:00
openstackgerritTzu-Mainn Chen proposed openstack/ironic master: Allow node owners to administer associated ports  https://review.opendev.org/69525519:05
TheJulia"zuul ate the jobs"19:09
*** tzumainn has quit IRC19:12
* TheJulia sighs as everything re-queues in zuul19:37
*** igordc has joined #openstack-ironic19:39
*** gmann is now known as gmann_afk19:40
*** dsneddon has quit IRC19:58
*** bobmel has joined #openstack-ironic20:01
*** tzumainn has joined #openstack-ironic20:02
*** bobmel has quit IRC20:05
*** dsneddon has joined #openstack-ironic20:14
*** rcernin has joined #openstack-ironic20:16
*** pcaruana has quit IRC20:16
*** dsneddon has quit IRC20:19
*** hjensas has joined #openstack-ironic20:27
*** Goneri has quit IRC20:38
*** dsneddon has joined #openstack-ironic20:40
*** dsneddon has quit IRC20:40
*** bobmel has joined #openstack-ironic20:41
*** naved001 has quit IRC20:44
*** naved001 has joined #openstack-ironic20:44
*** naved001 has quit IRC21:01
*** naved001 has joined #openstack-ironic21:01
*** jtomasek has quit IRC21:02
*** goldyfruit_ has quit IRC21:24
*** goldyfruit_ has joined #openstack-ironic21:30
*** dsneddon has joined #openstack-ironic21:32
*** bfournie has quit IRC21:57
*** dhellmann has quit IRC22:13
*** dhellmann has joined #openstack-ironic22:32
*** bfournie has joined #openstack-ironic22:54
*** gmann_afk is now known as gmann23:06
openstackgerritMerged openstack/ironic-python-agent master: Stop the API so rescue can exit and succeed  https://review.opendev.org/69784423:10
*** rh-jelabarre has quit IRC23:15
*** goldyfruit_ has quit IRC23:30
*** Goneri has joined #openstack-ironic23:35
*** Goneri has quit IRC23:40

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!