| *** dims has quit IRC | 00:02 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Update tox config to use latest upgrades https://review.openstack.org/47702 | 00:03 |
|---|---|---|
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Comply with pep440 https://review.openstack.org/46996 | 00:03 |
| *** dims has joined #openstack-infra | 00:04 | |
| *** sarob has joined #openstack-infra | 00:04 | |
| *** sarob has quit IRC | 00:08 | |
| *** dcramer_ has quit IRC | 00:13 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Rework run_shell_command https://review.openstack.org/42337 | 00:28 |
| *** jergerber has quit IRC | 00:34 | |
| *** senk has joined #openstack-infra | 00:35 | |
| *** senk has quit IRC | 00:39 | |
| *** sarob has joined #openstack-infra | 00:39 | |
| *** hub_cap has left #openstack-infra | 00:41 | |
| *** hub_cap has joined #openstack-infra | 00:41 | |
| *** sarob has quit IRC | 00:44 | |
| *** dcramer_ has joined #openstack-infra | 00:54 | |
| *** CaptTofu has quit IRC | 00:55 | |
| *** yongli has joined #openstack-infra | 00:56 | |
| mordred | fungi: wow. none of us caught this: https://review.openstack.org/#/c/40498/2/modules/openstack_project/templates/review.projects.yaml.erb | 01:12 |
| *** Daviey has quit IRC | 01:12 | |
| clarkb | (on my phone) what borked? | 01:13 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix a cargo-culting copy and paste error https://review.openstack.org/47703 | 01:13 |
| mordred | clarkb: adding oslo.version - the projects.yaml entry was just for a second oslo.config | 01:14 |
| mordred | clarkb: all four of us looked at the patch too :) | 01:15 |
| *** ryanpetrello has joined #openstack-infra | 01:16 | |
| Alex_Gaynor | FYI: dstufft and I are investigating the weird issue with timeouts on installing thrift | 01:21 |
| *** senk has joined #openstack-infra | 01:36 | |
| *** sarob has joined #openstack-infra | 01:40 | |
| *** senk has quit IRC | 01:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix a cargo-culting copy and paste error https://review.openstack.org/47703 | 01:42 |
| fungi | mordred: yeah, that was well-hidden | 01:42 |
| *** SlickNik_ has quit IRC | 01:44 | |
| *** sarob has quit IRC | 01:45 | |
| *** dcramer_ has quit IRC | 01:53 | |
| *** ryanpetrello has quit IRC | 01:55 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fix dependencies for activity-dev https://review.openstack.org/47708 | 01:58 |
| *** ericw has joined #openstack-infra | 02:00 | |
| *** sarob has joined #openstack-infra | 02:03 | |
| *** yaguang has joined #openstack-infra | 02:04 | |
| *** sarob has quit IRC | 02:07 | |
| *** senk has joined #openstack-infra | 02:37 | |
| *** sarob has joined #openstack-infra | 02:40 | |
| *** senk has quit IRC | 02:41 | |
| *** sarob has quit IRC | 02:45 | |
| *** ericw has quit IRC | 03:00 | |
| *** nati_ueno has joined #openstack-infra | 03:19 | |
| *** CaptTofu has joined #openstack-infra | 03:19 | |
| *** CaptTofu has quit IRC | 03:22 | |
| *** CaptTofu has joined #openstack-infra | 03:23 | |
| *** michchap has joined #openstack-infra | 03:31 | |
| *** ericw has joined #openstack-infra | 03:35 | |
| *** sdake has quit IRC | 03:36 | |
| *** michchap has quit IRC | 03:40 | |
| *** sarob has joined #openstack-infra | 03:41 | |
| *** rcleere has joined #openstack-infra | 03:46 | |
| *** sarob has quit IRC | 03:46 | |
| *** nati_ueno has quit IRC | 03:48 | |
| *** CaptTofu_ has joined #openstack-infra | 03:54 | |
| *** CaptTofu has quit IRC | 03:57 | |
| pleia2 | Ryan_Lane: happen to be here? | 04:13 |
| Ryan_Lane | yep | 04:14 |
| Ryan_Lane | hm | 04:14 |
| Ryan_Lane | I downloaded the image and ran it | 04:14 |
| Ryan_Lane | no network timeout :( | 04:14 |
| pleia2 | hm | 04:15 |
| pleia2 | sorry for testing this so late | 04:15 |
| Ryan_Lane | no worries | 04:15 |
| Ryan_Lane | let me try with a different virtual mac | 04:15 |
| pleia2 | thoughts on how to debug? | 04:15 |
| Ryan_Lane | ah, that seems to cause an issue | 04:15 |
| pleia2 | ok | 04:15 |
| Ryan_Lane | yep.... | 04:15 |
| Ryan_Lane | hmmm | 04:15 |
| Ryan_Lane | I wonder if the network is somehow dependent on the mac | 04:16 |
| Ryan_Lane | does the image boot eventually? | 04:16 |
| pleia2 | it boots | 04:16 |
| Ryan_Lane | just no network? | 04:16 |
| pleia2 | yeah | 04:16 |
| Ryan_Lane | ok, let me check network settings in the image when it boots | 04:16 |
| pleia2 | and takes a bit since it waits for the network timeout | 04:16 |
| Ryan_Lane | yep | 04:16 |
| Ryan_Lane | -_- | 04:17 |
| Ryan_Lane | I'm an idiot | 04:17 |
| Ryan_Lane | resolv.conf for sure is wrong | 04:18 |
| pleia2 | ah, udev is showing it thinks it has a eth0 (but just a eth1) | 04:18 |
| pleia2 | so everything is failing over to the actual eth1, which nothing is configured on | 04:18 |
| Ryan_Lane | hm. I wonder why there's no eth0... | 04:20 |
| Ryan_Lane | dmesg says its link isn't ready | 04:20 |
| pleia2 | I deleted eth0 from /etc/udev/rules.d/70-persistent-net.rules and it's ok! | 04:21 |
| Ryan_Lane | huh. weird | 04:21 |
| pleia2 | so it saved the mac of the eth0 from your thing there, when booting on other hardware it borked | 04:21 |
| Ryan_Lane | heh | 04:21 |
| pleia2 | added an eth1 to compensate | 04:21 |
| pleia2 | poorly :) | 04:21 |
| Ryan_Lane | ok. I'm going to fix the image and re-upload | 04:21 |
| pleia2 | cool | 04:21 |
| Ryan_Lane | for your local image, since you don't want to wait... | 04:21 |
| Ryan_Lane | the other broken thing is /etc/resolv.conf | 04:22 |
| Ryan_Lane | it's pointing at something internal to wikimedia labs | 04:22 |
| pleia2 | yeah | 04:22 |
| Ryan_Lane | I created everything via vmbuilder | 04:22 |
| pleia2 | want to just point it at 8.8.8.8 ? | 04:22 |
| Ryan_Lane | yep | 04:22 |
| pleia2 | cool | 04:22 |
| Ryan_Lane | ok, rebooting to ensure things work right | 04:23 |
| Ryan_Lane | hm. I wonder if I can make vmbuilder output a 32 bit | 04:23 |
| Ryan_Lane | it wouldn't help to have a 32 bit image, just in case | 04:24 |
| Ryan_Lane | err | 04:24 |
| Ryan_Lane | wouldn't hurt | 04:24 |
| Ryan_Lane | yep. arch = i386 | 04:25 |
| pleia2 | cool | 04:25 |
| Ryan_Lane | hm. resolv.conf is being reset somehow | 04:27 |
| pleia2 | foul thing | 04:27 |
| Ryan_Lane | stupid resolvconf | 04:27 |
| pleia2 | yeah, just found the listings myself | 04:28 |
| Ryan_Lane | where's it being set? | 04:28 |
| Ryan_Lane | I always have puppet manage this | 04:28 |
| Ryan_Lane | ah | 04:28 |
| pleia2 | /etc/resolvconf/resolvconf.d/tail and original | 04:29 |
| * Ryan_Lane nods | 04:29 | |
| pleia2 | er resolv.conf.d | 04:29 |
| pleia2 | but you'll find it | 04:29 |
| Ryan_Lane | yep :) | 04:29 |
| Ryan_Lane | this is what I get for trying to be smart and automate things :D | 04:29 |
| pleia2 | hehe | 04:29 |
| Ryan_Lane | much better. ok. going to re-export/upload | 04:30 |
| pleia2 | hmm, trying to get to horizon | 04:31 |
| pleia2 | I assume the procedure will be boot, run rejoin-stack.sh? | 04:31 |
| Ryan_Lane | boot, run stack.sh, then rejoin-stack.sh | 04:32 |
| Ryan_Lane | I ran unstack.sh | 04:32 |
| pleia2 | ok | 04:32 |
| Ryan_Lane | so that people would get a chance to run stack.sh and see what it's doing | 04:32 |
| * pleia2 nods | 04:32 | |
| *** anteaya has joined #openstack-infra | 04:35 | |
| pleia2 | hey anteaya! | 04:36 |
| anteaya | hey there | 04:36 |
| Ryan_Lane | howdy | 04:36 |
| anteaya | heya Ryan_Lane | 04:36 |
| Ryan_Lane | btw, the ova/ovf is set to 1024GB ram, but that can easily be changed if we want the vm to be 2GB | 04:37 |
| Ryan_Lane | that's MB, of course | 04:38 |
| pleia2 | hehe | 04:39 |
| pleia2 | so if we're just running cirros images, 1024 should be ok | 04:39 |
| pleia2 | and that's my plan right now | 04:39 |
| * Ryan_Lane nods | 04:40 | |
| *** sarob has joined #openstack-infra | 04:42 | |
| pleia2 | the 127.0.0.1 addresses aren't working for me | 04:42 |
| anteaya | some tests won't run if we have the ram at 1024 | 04:42 |
| anteaya | but some project's test suites should be okay | 04:42 |
| pleia2 | I wasn't planning on running tests, just some basic stuff | 04:42 |
| anteaya | glance and cinder should | 04:42 |
| anteaya | oh okay | 04:43 |
| pleia2 | if anyone actually wants to do development and tests and such moving forward, they should create a new image when they get home | 04:43 |
| Ryan_Lane | pleia2: which 127.0.0.1 ones aren't? | 04:44 |
| Ryan_Lane | the port forwards? | 04:44 |
| pleia2 | Ryan_Lane: yeah | 04:44 |
| Ryan_Lane | can you see if your image has them configured under the network? | 04:45 |
| Ryan_Lane | ssh -p 2222 stack@127.0.0.1 | 04:45 |
| pleia2 | ah, no | 04:45 |
| Ryan_Lane | oh. weird | 04:45 |
| Ryan_Lane | it should have been exported with the ova | 04:46 |
| Ryan_Lane | it works for my import | 04:46 |
| *** sarob has quit IRC | 04:46 | |
| pleia2 | I loaded up the vmdk | 04:46 |
| Ryan_Lane | ooohhh | 04:46 |
| pleia2 | I'll try with ova :) | 04:46 |
| Ryan_Lane | yeah, the ova/ovf is what has the settings | 04:47 |
| anteaya | I too have been using the vmdk | 04:47 |
| Ryan_Lane | ova is vmdk + settings | 04:47 |
| pleia2 | anteaya: you can add them manually to virtualbox | 04:47 |
| pleia2 | ah | 04:47 |
| Ryan_Lane | I have the vmdk just in case the ova doesn't work | 04:47 |
| anteaya | but I have to download the ova file, correct? | 04:47 |
| pleia2 | anteaya: nah | 04:47 |
| Ryan_Lane | actually, you can use the ovf | 04:48 |
| Ryan_Lane | or you can do your port forwards manuall\y | 04:48 |
| pleia2 | ovf may be easier | 04:48 |
| pleia2 | anteaya: download this file: https://wikitech.wikimedia.org/devstack/devstack.ovf | 04:48 |
| pleia2 | it's very small | 04:48 |
| anteaya | got it | 04:49 |
| pleia2 | Ryan_Lane: hm, how do you apply an ovf to vmdk? | 04:49 |
| Ryan_Lane | file -> import appliance | 04:49 |
| Ryan_Lane | you can specify the ova or the ovf that way | 04:50 |
| pleia2 | sweet | 04:50 |
| anteaya | so if I import appliance for the .ovf and then launch that machine | 04:54 |
| pleia2 | yeah | 04:55 |
| anteaya | I need to rejoin-stack? | 04:55 |
| *** hashar has quit IRC | 04:55 | |
| Ryan_Lane | you'll need to stack.sh, then rejoin-stack.sh | 04:55 |
| anteaya | if I run stack.sh, why do I need to run rejoin-stack? | 04:55 |
| Ryan_Lane | it doesn't automatically connect you to the screen when you run stack.sh | 04:56 |
| Ryan_Lane | I think it used to. not sure why it doesn't anymore | 04:56 |
| anteaya | I usuing just run `screen -r stack` | 04:57 |
| anteaya | usually | 04:57 |
| anteaya | that connects me to the screen | 04:57 |
| * Ryan_Lane nods | 04:57 | |
| anteaya | it appears the script serves the same function | 04:58 |
| anteaya | I have imported the .ovf file, launched that vm and run stack.sh | 04:59 |
| anteaya | should I be able to connect to http://127.0.0.1:8000/ via my browser at this point? | 05:00 |
| anteaya | because if that is the expectation, I can not do so | 05:01 |
| pleia2 | has stack.sh finished? | 05:02 |
| Ryan_Lane | hm. did I write 8000? | 05:02 |
| Ryan_Lane | try 8080 | 05:02 |
| pleia2 | ah yes, it should be 8080 | 05:03 |
| pleia2 | elizabeth@r4e1:~$ nc 127.0.0.1 2222 | 05:05 |
| pleia2 | SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1 | 05:05 |
| pleia2 | \o/ | 05:05 |
| Ryan_Lane | :) | 05:05 |
| anteaya | http://127.0.0.1:8080/ works | 05:06 |
| Ryan_Lane | cool | 05:06 |
| anteaya | anita@anita-hp:~$ nc 127.0.0.1 2222 | 05:06 |
| anteaya | SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1 | 05:06 |
| pleia2 | hooray, all good | 05:07 |
| pleia2 | I am not going to put passwords on this doc, instead I'll have them on a whiteboard or similar at the event | 05:08 |
| Ryan_Lane | ok, fixed 64 bit images are up | 05:08 |
| pleia2 | great | 05:09 |
| *** nati_ueno has joined #openstack-infra | 05:10 | |
| anteaya | I have launched an instance via the gui | 05:11 |
| anteaya | anything else? | 05:11 |
| pleia2 | that's it :) | 05:11 |
| anteaya | great | 05:12 |
| anteaya | success | 05:12 |
| anteaya | plus it is before midnight | 05:12 |
| anteaya | woohoo | 05:12 |
| pleia2 | yes! you should get some rest | 05:12 |
| anteaya | I will do so | 05:13 |
| anteaya | thanks Ryan_Lane | 05:13 |
| pleia2 | Ryan_Lane: anteaya is sick and I have a sprained ankle (seattle was hard on us!) so I'm thinking we'll get an uber down (flat rate to palo alto), what are yourplans? from sf | 05:13 |
| anteaya | thanks pleia2 | 05:13 |
| pleia2 | anteaya: thanks for your patience :) | 05:14 |
| anteaya | ha ha ha | 05:14 |
| anteaya | :D | 05:14 |
| *** nati_ueno has quit IRC | 05:14 | |
| Ryan_Lane | pleia2: ah. I was going to take caltrain | 05:15 |
| Ryan_Lane | uber is relatively expensive from sf to palo alto | 05:16 |
| pleia2 | uberx is $90 | 05:16 |
| Ryan_Lane | oh, really? | 05:16 |
| pleia2 | regular is $120 | 05:16 |
| Ryan_Lane | hm. where are you two going to catch it? maybe I can meet you and we can split 3 ways | 05:17 |
| pleia2 | I'm thinking around 10:30 | 05:17 |
| pleia2 | gives us a little time in case there is traffic | 05:17 |
| * Ryan_Lane nods | 05:17 | |
| Ryan_Lane | at your building? | 05:18 |
| pleia2 | yeah | 05:18 |
| Ryan_Lane | cool. I'll meet there/then | 05:19 |
| pleia2 | great :) | 05:19 |
| anteaya | sounds good to me too | 05:19 |
| anteaya | your doorman is so friendly, I'll just wait in the lobby | 05:19 |
| pleia2 | yeah, he's awesome :) | 05:20 |
| anteaya | okay I'm off for the night | 05:20 |
| anteaya | see you both tomorrow | 05:20 |
| *** anteaya has quit IRC | 05:20 | |
| Ryan_Lane | 32 bit is uploading | 05:21 |
| Ryan_Lane | so, I think we're good to go from the image front for tomorrow | 05:21 |
| pleia2 | yeah | 05:21 |
| pleia2 | I added some notes to the documentation doc if you want to have a look before I prnit | 05:22 |
| Ryan_Lane | sure. one sec | 05:22 |
| Ryan_Lane | looks good | 05:23 |
| pleia2 | thanks | 05:25 |
| *** annegentle has quit IRC | 05:25 | |
| *** annegentle has joined #openstack-infra | 05:27 | |
| *** sarob has joined #openstack-infra | 05:36 | |
| *** sarob has quit IRC | 05:40 | |
| *** sarob has joined #openstack-infra | 06:04 | |
| *** annegentle has quit IRC | 06:09 | |
| *** annegentle has joined #openstack-infra | 06:12 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:30 | |
| *** Ryan_Lane has quit IRC | 06:58 | |
| *** Ryan_Lane has joined #openstack-infra | 07:00 | |
| lifeless | jeblair: ack | 07:02 |
| lifeless | jeblair: so, right now we have chicken and egg: we can't sensibly ask for hardware because we don't have the infra in place to use it; we have blocked on doing the infra because we didn't have hardware. | 07:02 |
| lifeless | jeblair: I want to break the deadlock, get something properly integrated in place and then hunt down multi-vendor sponsorship to get redundancy like we have with clouds. | 07:03 |
| lifeless | jeblair: as for me and lotteries :P... short term anyone @ HP in Monty's team has, or can have, access to the IPMI creds for the machines; the actual cloud deployed there is tripleo and thus anyone can fix bugs | 07:04 |
| lifeless | jeblair: and the various tooling we come up with will belong in openstack[-infra at your discretion] so also fixable by anyone. | 07:05 |
| lifeless | jeblair: we have no formal SLA for fixing the hardware that I know of, so I can't pass one onto infra by proxy; I will investigate. | 07:05 |
| lifeless | jeblair: I also need to investigate the possibility of doing check jobs (vs gate jobs) from HP IT security folk. | 07:06 |
| lifeless | or, rather - I might need to investigate that, I haven't walked through the last-little-detail of the design yet. | 07:06 |
| lifeless | jeblair: I will get a mail thread rolling around this next week. | 07:07 |
| lifeless | jeblair: thanks for the docs link. | 07:07 |
| *** Ryan_Lane has quit IRC | 07:10 | |
| *** thomasbiege has joined #openstack-infra | 07:19 | |
| *** thomasbiege has quit IRC | 07:24 | |
| *** yolanda has joined #openstack-infra | 07:40 | |
| *** senk has joined #openstack-infra | 07:41 | |
| *** sarob has quit IRC | 07:41 | |
| *** sarob has joined #openstack-infra | 07:41 | |
| *** s1rp has quit IRC | 07:44 | |
| *** s1rp has joined #openstack-infra | 07:45 | |
| *** senk has quit IRC | 07:45 | |
| *** sarob has quit IRC | 07:46 | |
| *** thomasbiege has joined #openstack-infra | 08:10 | |
| *** senk has joined #openstack-infra | 08:11 | |
| *** thomasbiege has quit IRC | 08:15 | |
| *** senk has quit IRC | 08:17 | |
| *** sarob has joined #openstack-infra | 08:53 | |
| *** sarob has quit IRC | 08:58 | |
| *** boris-42 has joined #openstack-infra | 09:01 | |
| *** kpavel has joined #openstack-infra | 09:19 | |
| *** portante has quit IRC | 09:29 | |
| *** matty_dubs|gone has quit IRC | 09:29 | |
| *** portante has joined #openstack-infra | 09:30 | |
| *** matty_dubs|gone has joined #openstack-infra | 09:30 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/pbr: Fix git-dir parameter placement https://review.openstack.org/47736 | 09:41 |
| *** SergeyLukjanov has quit IRC | 09:46 | |
| *** yaguang has quit IRC | 10:13 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:28 | |
| *** mkoderer has quit IRC | 10:58 | |
| *** DinaBelova has joined #openstack-infra | 10:59 | |
| *** DinaBelova has quit IRC | 11:04 | |
| *** SergeyLukjanov has quit IRC | 11:04 | |
| *** ianw has quit IRC | 11:20 | |
| *** boris-42 has quit IRC | 11:20 | |
| *** cthulhup has joined #openstack-infra | 11:53 | |
| *** cthulhup has quit IRC | 11:59 | |
| *** cthulhup has joined #openstack-infra | 11:59 | |
| *** michchap has joined #openstack-infra | 12:15 | |
| *** michchap_ has joined #openstack-infra | 12:16 | |
| *** michchap has quit IRC | 12:20 | |
| *** DinaBelova has joined #openstack-infra | 12:38 | |
| *** fifieldt has joined #openstack-infra | 12:48 | |
| *** mkoderer_ has joined #openstack-infra | 12:54 | |
| *** CaptTofu_ has quit IRC | 13:01 | |
| *** CaptTofu has joined #openstack-infra | 13:01 | |
| *** cthulhup has quit IRC | 13:02 | |
| *** DinaBelova has quit IRC | 13:02 | |
| *** flaper87|afk is now known as flaper87 | 13:22 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:38 | |
| *** dims has quit IRC | 13:45 | |
| *** fifieldt has quit IRC | 13:48 | |
| *** cthulhup has joined #openstack-infra | 14:09 | |
| *** CaptTofu has quit IRC | 14:17 | |
| *** CaptTofu has joined #openstack-infra | 14:17 | |
| *** CaptTofu has quit IRC | 14:21 | |
| *** CaptTofu has joined #openstack-infra | 14:21 | |
| *** kpavel has quit IRC | 14:40 | |
| mordred | lifeless: you don't have to investigate that possibility | 14:50 |
| mordred | lifeless: that rack is quite explictly considered hostile | 14:50 |
| mordred | and the networking around it was designed specifically with the intent that untrusted code from the internet would run onit | 14:51 |
| *** CaptTofu has quit IRC | 14:52 | |
| *** thomasm has joined #openstack-infra | 14:52 | |
| *** CaptTofu has joined #openstack-infra | 14:52 | |
| *** blamar has quit IRC | 14:52 | |
| *** dizquierdo has joined #openstack-infra | 14:54 | |
| *** cody-somerville has joined #openstack-infra | 15:06 | |
| *** cthulhup has quit IRC | 15:07 | |
| *** cthulhup has joined #openstack-infra | 15:07 | |
| *** dcramer_ has joined #openstack-infra | 15:10 | |
| *** SergeyLukjanov has quit IRC | 15:10 | |
| *** cthulhup has quit IRC | 15:12 | |
| *** cody-somerville has quit IRC | 15:21 | |
| *** CaptTofu has quit IRC | 15:26 | |
| *** CaptTofu has joined #openstack-infra | 15:27 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:27 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack/requirements: Debian needs compatibility with SQLAlchemy 0.8.x https://review.openstack.org/47745 | 15:33 |
| Alex_Gaynor | fungi: it's my understanding that various projects do not actually work with 0.8 | 15:33 |
| fungi | Alex_Gaynor: yes, that ^ is a test to see what breaks | 15:34 |
| fungi | which is why it's wuip | 15:34 |
| fungi | er, wip | 15:34 |
| Alex_Gaynor | oh hey, didn't notice that | 15:35 |
| Alex_Gaynor | Oh, second email. | 15:35 |
| sdague | yeh, heat blows up last time someone proposed that | 15:35 |
| sdague | however.... right now, the gate is bust in other ways | 15:35 |
| clarkb | sdague: is it? there was one email but no details and zuul status lacks significant red. also early sunday morning | 15:37 |
| fungi | yeah, i was basically just pointing out to zigo that our global requirements list has capped sqlalchemy to 0.7.x only for over a year, and that there's no "back and forth" as he suggests | 15:37 |
| *** pabelanger has quit IRC | 15:38 | |
| fungi | garyk's e-mail about broken devstack gate was also tagged [nova] so maybe it's a nova-specific concern? | 15:39 |
| sdague | it's not | 15:39 |
| sdague | but it would have been nice if garyk had provided more details | 15:39 |
| sdague | I'm most of the way through an email with some analysis | 15:40 |
| fungi | more details would certainly be nice. as clarkb points out, zuul's status page is lots of green and/or in-progress | 15:40 |
| sdague | yeh, the fails are in the devstack jobs, which run about the same length of time | 15:42 |
| sdague | so they'll error really fast before the job is killed, as for everything but neutron they are the longest jobs still | 15:43 |
| clarkb | fungi: I think zigo is referring to his first request to use 0.8 re back and forth. arguably we should have worked on this long ago when we first pinned sqlalchemy. these are the danger of pinning | 15:43 |
| sdague | clarkb: we pinned that around g2 | 15:43 |
| sdague | fungi: ok, email out | 15:44 |
| fungi | around g2 of folsom maybe? | 15:44 |
| sdague | hmmm... actually it might have been end of folsom | 15:44 |
| fungi | because there has never been a point in the history of the openstack/requirements repository where we had an sqlalchemy version spec which would have alowed 0.8 | 15:45 |
| sdague | fungi: so the short version of the devstack fail is I don't really know the issue, but we are failing to launch guests | 15:45 |
| fungi | k | 15:45 |
| sdague | and we also seem to be using a very new cloud archive version of libvirt since midnight | 15:45 |
| fungi | oh, awesome | 15:45 |
| sdague | so my suspicion is that's related | 15:45 |
| fungi | and yes, we did just start supporting cloud archive back on friday. maybe this is the first time the slaves/images have updated to the cloud archive libvirt | 15:46 |
| sdague | yes, that makes sense | 15:46 |
| clarkb | we initially pinned due to an internal sqlalchemy api change around exceptions iirc and we did that when 0.8 was released | 15:47 |
| jeblair | i'll check out timestamps/logs from nodepool to explore that idea | 15:47 |
| sdague | jeblair: as far as I can tell no integrated project has landed a merge since midnight | 15:47 |
| clarkb | but now there are other problems like migrate which is why we are hosting that project now | 15:47 |
| *** boris-42 has joined #openstack-infra | 15:48 | |
| sdague | and there is a lot of jenkins -1 activity on - https://review.openstack.org/#/q/status:open+project:openstack/nova,n,z | 15:48 |
| sdague | the fails all look about the same | 15:48 |
| jeblair | the current images used are 13 hours old | 15:48 |
| jeblair | the previous images are 61 hours old | 15:49 |
| *** dizquierdo has quit IRC | 15:49 | |
| jeblair | if we want to fix this on the infra side (instead of making things work with the cloud-archive libvirt).... | 15:49 |
| jeblair | we can delete the current images and nodepool will revert to using the previous 61 hour old images | 15:50 |
| jeblair | and then merge a change to revert the use of cloud archive, so that the next image build won't use it | 15:50 |
| jeblair | sdague: i haven't looked into the actual error at all; what approach would you recommend? | 15:51 |
| sdague | so, n-cpu ends at a weird time | 15:51 |
| sdague | which means I think we crashed it | 15:51 |
| sdague | so my suggestion is to revert, then provide for some ability to figure out what's going on manually next week with the cloud-archive libvirt | 15:52 |
| fungi | (and then suggest that the cloud archive addition be resubmitted after things support newer libvirt or include a package pin to the precise version of the libvirt package) | 15:52 |
| * fungi proposes revert | 15:52 | |
| mordred | oh! everyone is up | 15:52 |
| fungi | mordred: only when everything breaks | 15:53 |
| jeblair | fungi: thanks; i'll delete the current nodepool images | 15:53 |
| * clarkb plays devil's advocate and points at sqlalchemy for reasons why we should fix it properly | 15:53 | |
| jeblair | sdague: we can test easily by proposing a change to devstack-gate to install cloud-archive | 15:53 |
| sdague | jeblair: cool, sounds good | 15:53 |
| mordred | wow. what fun scrollback | 15:55 |
| mordred | jeblair: ++ | 15:55 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Revert "Use Ubuntu Cloud Archive on precise nodes" https://review.openstack.org/47748 | 15:55 |
| fungi | gong ^ | 15:55 |
| mordred | I do agree with clarkb's sentiment, although I htink rollback is the best choice right now | 15:55 |
| mordred | I think that getting working on cloud-archive libvirt should be a high priority - (whether that means fixing nova or fixing cloud-archive) | 15:56 |
| jeblair | clarkb, fungi, mordred: http://paste.openstack.org/show/47358/ | 15:56 |
| jeblair | nodepool commands i used ^ | 15:56 |
| mordred | jeblair: thanks | 15:56 |
| fungi | oh, neat! | 15:57 |
| fungi | i hadn't had a chance to try out the new nodepool subcommands yet | 15:57 |
| jeblair | it doesn't flag the current image; i should have it do that. i'm just looking for the lowest value in the age column to determine the 'current' image | 15:57 |
| sdague | the fact that n-cpu just stops makes me think our python-libvirt crashed in a terrible way | 15:57 |
| fungi | makes sense | 15:57 |
| jeblair | also, i don't know why there is more than one old image. could be a bug. | 15:57 |
| jeblair | (but i'm not complaining; actually, i'm now thinking we should have nodepool keep images around for a week. because why not.) | 15:58 |
| fungi | disk is cheap ;) | 15:58 |
| fungi | (especially when we don't get charged for it) | 15:59 |
| sdague | ok, time to deal with house hold activities. This will be top of my queue on monday. | 16:00 |
| clarkb | the images are relatively small too | 16:00 |
| * fungi is semi-afk. supposed to be baking chris a bday cake right now | 16:00 | |
| *** CaptTofu has quit IRC | 16:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Use Ubuntu Cloud Archive on precise nodes" https://review.openstack.org/47748 | 16:01 |
| *** CaptTofu has joined #openstack-infra | 16:01 | |
| jeblair | gong ^ | 16:01 |
| clarkb | :) | 16:01 |
| jeblair | clarkb: ah, i need to restart nodepool to pick up your boot timeout change | 16:02 |
| jeblair | it's been running since sept 10, so the running code doesn't support the option | 16:02 |
| jeblair | done; hopefully the building/deleting lines on that graph will look a bit better now. | 16:04 |
| clarkb | cool | 16:04 |
| Alex_Gaynor | So should stuff in theory return to working now? | 16:08 |
| clarkb | slowly, any existing VMs need to be cycled out | 16:11 |
| jeblair | i'm going ahead and deleting the 'ready' vms, so that should speed that up a bit | 16:11 |
| jeblair | (it's mostly the case where a few might stick around and get used hours later that i want to avoid) | 16:11 |
| jeblair | also, YAY! i can see a failure on the new zuul status screen! | 16:12 |
| Alex_Gaynor | clarkb: Should stuff be recheck'd yet, or wait? | 16:13 |
| clarkb | I would wait for jeblair to finish deleting VMs | 16:14 |
| jeblair | i should really make that "nargs='+'" .... | 16:15 |
| *** Ryan_Lane has joined #openstack-infra | 16:18 | |
| jeblair | done | 16:20 |
| *** SergeyLukjanov has quit IRC | 16:28 | |
| openstackgerrit | Rongze Zhu proposed a change to openstack-infra/gitdm: Update unitedstack domain map https://review.openstack.org/47754 | 16:40 |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Update unitedstack domain map https://review.openstack.org/47754 | 16:45 |
| *** Ryan_Lane has quit IRC | 16:46 | |
| *** ericw has quit IRC | 17:00 | |
| openstackgerrit | A change was merged to openstack/requirements: Added pypy env to tox https://review.openstack.org/47697 | 17:07 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Check and gate requirements against PyPy https://review.openstack.org/47756 | 17:09 |
| *** ericw has joined #openstack-infra | 17:16 | |
| openstackgerrit | A change was merged to openstack/requirements: pycadf 0.1.8 broke oslo https://review.openstack.org/47380 | 17:21 |
| *** ericw has quit IRC | 17:28 | |
| *** ericw has joined #openstack-infra | 17:30 | |
| *** CaptTofu has quit IRC | 17:39 | |
| *** CaptTofu has joined #openstack-infra | 17:40 | |
| *** CaptTofu has quit IRC | 17:42 | |
| *** CaptTofu has joined #openstack-infra | 17:42 | |
| *** sdake has joined #openstack-infra | 17:48 | |
| *** ericw has quit IRC | 17:50 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:51 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add doc build and publishing to tripleo-incubator https://review.openstack.org/47759 | 18:01 |
| *** CaptTofu has quit IRC | 18:02 | |
| *** pabelanger has joined #openstack-infra | 18:03 | |
| *** sdake has quit IRC | 18:07 | |
| *** CaptTofu has joined #openstack-infra | 18:07 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add doc build and publishing to tripleo-incubator https://review.openstack.org/47759 | 18:09 |
| *** ericw has joined #openstack-infra | 18:14 | |
| *** ericw has quit IRC | 18:15 | |
| *** thomasbiege has joined #openstack-infra | 18:17 | |
| *** zehicle has quit IRC | 18:22 | |
| *** zehicle has joined #openstack-infra | 18:23 | |
| lifeless | mordred: lets come back to that another time | 18:25 |
| *** dims has joined #openstack-infra | 18:27 | |
| *** Ryan_Lane has joined #openstack-infra | 18:42 | |
| *** thomasbiege has quit IRC | 18:49 | |
| *** dcramer_ has quit IRC | 18:57 | |
| *** dcramer_ has joined #openstack-infra | 19:09 | |
| *** Ryan_Lane has quit IRC | 19:10 | |
| *** sarob has joined #openstack-infra | 19:12 | |
| *** adam_g has quit IRC | 19:15 | |
| *** sarob has quit IRC | 19:17 | |
| *** adam_g has joined #openstack-infra | 19:19 | |
| *** adam_g has quit IRC | 19:19 | |
| *** adam_g has joined #openstack-infra | 19:19 | |
| *** Ryan_Lane has joined #openstack-infra | 19:20 | |
| *** anteaya has joined #openstack-infra | 19:27 | |
| *** Ryan_Lane has quit IRC | 19:32 | |
| *** dcramer_ has quit IRC | 19:32 | |
| *** Ryan_Lane has joined #openstack-infra | 19:35 | |
| *** sarob has joined #openstack-infra | 19:43 | |
| *** sarob has quit IRC | 19:51 | |
| *** CaptTofu has quit IRC | 19:53 | |
| *** CaptTofu has joined #openstack-infra | 19:54 | |
| *** yolanda has quit IRC | 20:00 | |
| *** Daviey has joined #openstack-infra | 20:06 | |
| *** SergeyLukjanov has quit IRC | 20:13 | |
| *** ArxCruz has joined #openstack-infra | 20:16 | |
| *** sarob has joined #openstack-infra | 20:18 | |
| *** stevebaker has quit IRC | 20:20 | |
| *** stevebaker has joined #openstack-infra | 20:20 | |
| *** sarob has quit IRC | 20:22 | |
| *** zehicle has quit IRC | 20:27 | |
| *** ArxCruz has quit IRC | 20:29 | |
| *** zehicle has joined #openstack-infra | 20:31 | |
| *** briancline has quit IRC | 20:50 | |
| *** briancline has joined #openstack-infra | 20:50 | |
| *** flaper87 is now known as flaper87|afk | 20:51 | |
| *** Ryan_Lane has quit IRC | 20:53 | |
| *** flaper87|afk is now known as flaper87 | 20:58 | |
| *** Ryan_Lane has joined #openstack-infra | 20:58 | |
| *** Ryan_Lane has quit IRC | 20:59 | |
| sdague | fungi / jeblair: how do I get ahold of the base image that was built differently (with the cloud archive libvirt)? I'd like to experiment locally | 21:04 |
| *** dkehn has quit IRC | 21:07 | |
| *** dkehn has joined #openstack-infra | 21:10 | |
| *** sarob has joined #openstack-infra | 21:18 | |
| clarkb | sdague: I think the image that was in use was deleted (as there isn't another way to force nodepool to not use it) however if you use http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_node_devstack.sh and a copy of the infra config repo that doesn't have cloud archive reverted you should get the same thing | 21:19 |
| *** flaper87 is now known as flaper87|afk | 21:20 | |
| mordred | lifeless: I'd like to land a bunch of your docs patches, but they depend on https://review.openstack.org/#/c/45011/5 which has a -1 | 21:23 |
| mordred | lifeless: could you fix that up so that we can land the others too? | 21:23 |
| *** sarob has quit IRC | 21:23 | |
| clarkb | mordred: we can also make them all independent patches... | 21:24 |
| mordred | clarkb: yes. I'd be fine with either thing | 21:25 |
| *** zehicle has quit IRC | 21:26 | |
| *** Ryan_Lane has joined #openstack-infra | 21:27 | |
| Alex_Gaynor | mordred: do you understand the error here: https://review.openstack.org/#/c/47756/ I don't see why gate-requirements-pypy wouldn't be a thing | 21:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Add translation updates for Heat https://review.openstack.org/43991 | 21:29 |
| clarkb | Alex_Gaynor: you need to add the job in the JJB config as well | 21:31 |
| Alex_Gaynor | clarkb: yup, just realized it wasn't using python-jobs there, thanks. | 21:31 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Check and gate requirements against PyPy https://review.openstack.org/47756 | 21:32 |
| *** Ryan_Lane has quit IRC | 21:35 | |
| *** zehicle has joined #openstack-infra | 21:39 | |
| *** dkehn has quit IRC | 21:43 | |
| *** thomasm has quit IRC | 21:47 | |
| lifeless | mordred: yes, I shall | 21:47 |
| lifeless | mordred: I sorted out with jeblair what is needed, just haven't had time to execute yet. | 21:48 |
| lifeless | mordred: I may rebase things to put all the approved at the bottom | 21:48 |
| *** Ryan_Lane has joined #openstack-infra | 21:48 | |
| *** DennyZhang has joined #openstack-infra | 21:55 | |
| mordred | lifeless: k. cool | 22:00 |
| *** DennyZhang has quit IRC | 22:03 | |
| *** DennyZhang has joined #openstack-infra | 22:03 | |
| *** senk has joined #openstack-infra | 22:12 | |
| *** senk has quit IRC | 22:14 | |
| *** dkehn has joined #openstack-infra | 22:16 | |
| *** sarob has joined #openstack-infra | 22:19 | |
| *** DennyZhang has quit IRC | 22:19 | |
| *** sarob has quit IRC | 22:23 | |
| *** UtahDave has joined #openstack-infra | 22:26 | |
| *** sdake has joined #openstack-infra | 22:28 | |
| *** sdake has joined #openstack-infra | 22:28 | |
| *** DennyZhang has joined #openstack-infra | 22:28 | |
| *** zehicle has quit IRC | 22:28 | |
| *** dkehn has quit IRC | 22:31 | |
| *** dkehn has joined #openstack-infra | 22:33 | |
| *** jhesketh has joined #openstack-infra | 22:33 | |
| *** jhesketh__ has joined #openstack-infra | 22:34 | |
| *** senk has joined #openstack-infra | 22:35 | |
| *** zehicle has joined #openstack-infra | 22:42 | |
| *** ericw has joined #openstack-infra | 22:48 | |
| *** senk has quit IRC | 22:49 | |
| *** CaptTofu has quit IRC | 23:01 | |
| Ryan_Lane | :D | 23:03 |
| Ryan_Lane | pleia2: you must be in backscroll | 23:04 |
| anteaya | yes | 23:04 |
| Ryan_Lane | heh. was trying to presentation bomb | 23:04 |
| *** jhesketh has quit IRC | 23:04 | |
| *** jhesketh has joined #openstack-infra | 23:05 | |
| anteaya | and now the terminal is gone | 23:05 |
| zul | sdague: if you have ubuntu installed rough intstructions http://pastebin.ubuntu.com/6143417/ | 23:10 |
| *** zehicle has quit IRC | 23:15 | |
| *** sarob has joined #openstack-infra | 23:17 | |
| *** ericw has quit IRC | 23:19 | |
| *** boris-42 has quit IRC | 23:21 | |
| *** sarob has quit IRC | 23:22 | |
| *** DennyZhang has quit IRC | 23:23 | |
| *** annegentle has quit IRC | 23:25 | |
| *** ericw has joined #openstack-infra | 23:26 | |
| *** annegentle has joined #openstack-infra | 23:27 | |
| *** alexpilotti has quit IRC | 23:30 | |
| *** Ryan_Lane has quit IRC | 23:32 | |
| *** anteaya has quit IRC | 23:36 | |
| *** zehicle has joined #openstack-infra | 23:39 | |
| *** zehicle has quit IRC | 23:44 | |
| *** sarob has joined #openstack-infra | 23:48 | |
| *** ericw has quit IRC | 23:50 | |
| *** senk has joined #openstack-infra | 23:50 | |
| *** sarob has quit IRC | 23:53 | |
| *** senk has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!