| kjorgensen | the message saying "Starting check jobs" and the 'status_url' from /etc/zuul/zuul.conf | 00:00 |
|---|---|---|
| anteaya | okay | 00:00 |
| anteaya | and is the job currently stuck in the queue on your jenkins? | 00:01 |
| kjorgensen | well it doesn't even show in the jenkins web interface | 00:02 |
| *** dizquierdo has quit IRC | 00:02 | |
| anteaya | okay | 00:02 |
| anteaya | lets see if we can find out what the gearman server thinks is happening | 00:02 |
| anteaya | can you find logs for the gearman server? | 00:02 |
| kjorgensen | they are in /var/log/zuul/gearman-server.log right? | 00:03 |
| anteaya | to be honest I don't know | 00:04 |
| anteaya | I personally don't run these services | 00:04 |
| anteaya | I'm just really practiced at asking questions | 00:04 |
| *** tbachman has joined #openstack-infra | 00:04 | |
| anteaya | but if there is a file at that path, let's see what it says | 00:04 |
| *** chlong has quit IRC | 00:05 | |
| *** kzaitsev_mb has quit IRC | 00:06 | |
| *** rbrndt has quit IRC | 00:07 | |
| kjorgensen | ok :) yea i'm not 100% sure either but I assume it's the one. Strangely, that file is mostly empty except for a cryptic error message http://paste.openstack.org/show/496294/ | 00:08 |
| *** yamamoto has joined #openstack-infra | 00:08 | |
| anteaya | kjorgensen: I'm going to conclude that your gearman server is not running | 00:08 |
| anteaya | kjorgensen: are you able to assess if it is running or not? | 00:09 |
| anteaya | ah you are using python2.6 | 00:09 |
| anteaya | is there a reason you are using python2.6? | 00:09 |
| anteaya | could you possibly convinced to try running it with python2.7? | 00:09 |
| anteaya | and what version of gear are you running? | 00:10 |
| *** sflanigan has joined #openstack-infra | 00:11 | |
| *** sflanigan has joined #openstack-infra | 00:11 | |
| kjorgensen | haha yea we're kinda behind | 00:11 |
| kjorgensen | and by "kinda" I mean "very" | 00:11 |
| anteaya | current version of gear is 0.5.9 https://pypi.python.org/pypi/gear/0.5.9 | 00:12 |
| anteaya | well we may have found a potential issue | 00:12 |
| anteaya | good news is the path forward is very clear | 00:12 |
| anteaya | I'd say check all your versions and upgrade things then give it another shot | 00:13 |
| anteaya | also we have a puppet module: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/ | 00:13 |
| anteaya | that has instructions for how folks can set up their own ci | 00:13 |
| *** ddieterly has joined #openstack-infra | 00:13 | |
| anteaya | the use case for this module is that folks set up their own testing to interact with our gerrit | 00:14 |
| anteaya | but you can set up your own gerrit and have your ci point to that | 00:14 |
| *** vhosakot has joined #openstack-infra | 00:14 | |
| kjorgensen | so my installed versions are gear==0.5.9 and zuul==2.1.0 | 00:14 |
| anteaya | well that is the latest gear | 00:15 |
| asselin_ | kjorgensen, anteaya the published docs for that repo are here: http://docs.openstack.org/infra/openstackci/ | 00:15 |
| anteaya | and the latest zuul: http://git.openstack.org/cgit/openstack-infra/zuul/refs/ | 00:15 |
| anteaya | thanks asselin_ | 00:16 |
| anteaya | kjorgensen: any thoughts on trying it with a newer python release? | 00:16 |
| anteaya | asselin_: I missed you at summit | 00:16 |
| kjorgensen | anteaya: I can look into that | 00:16 |
| anteaya | kjorgensen: sounds like it might be worth investigating | 00:16 |
| asselin_ | yeah, unfortunately I couldn't make it | 00:16 |
| anteaya | :( | 00:17 |
| anteaya | asselin_: what python version do you run for your tooling? | 00:17 |
| asselin_ | 2.7 | 00:17 |
| anteaya | thanks | 00:17 |
| kjorgensen | anteaya: thanks for trying to help | 00:17 |
| anteaya | I always try | 00:18 |
| anteaya | :) | 00:18 |
| kjorgensen | we actually use chef for our CI deployments, I'm guessing there isn't docs to set up zuul with chef? ;) | 00:19 |
| anteaya | well our team uses puppet | 00:19 |
| bkero | You could probably adapt the puppet modules | 00:19 |
| anteaya | but chef does have cookbooks for openstack service deployment | 00:20 |
| anteaya | I don't know as they have written any cookbooks for deploying our infra tools though | 00:20 |
| anteaya | but you can look | 00:20 |
| anteaya | git.openstack.org | 00:20 |
| bkero | kjorgensen: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests | 00:20 |
| anteaya | is the home of all our repos | 00:20 |
| anteaya | clarkb: thank you this is a really good talk: https://www.youtube.com/watch?v=LRMBXnYzcP4 | 00:21 |
| asselin_ | pabelanger has some in ansible I believe..... | 00:21 |
| anteaya | I really enjoyed it | 00:21 |
| anteaya | dims: I recommend you taking a look at https://www.youtube.com/watch?v=LRMBXnYzcP4 and consider adding it to your blog post page | 00:21 |
| anteaya | I think it is a really well done talk | 00:21 |
| kjorgensen | thanks anteaya, yea I can look at the puppet code and try to adapt it | 00:22 |
| anteaya | kjorgensen: you are welcome to ask questions as you go along | 00:22 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add test and update documentation for mailer plugin https://review.openstack.org/312810 | 00:22 |
| *** dtantsur has joined #openstack-infra | 00:22 | |
| anteaya | many folks are in the same boat as yourself, so have different thoughts on how to get you to where you want to go | 00:23 |
| anteaya | kjorgensen: asselin_ has set up a number of ci systems, has helped a ton of folks to set up theirs and wrote the puppet-openstackci module and documentation | 00:23 |
| kjorgensen | you were saying that you don't think my gearman server was running, but I think it is, because I can run 'telnet localhost 4730' and then get the 'status' from my gearman server | 00:25 |
| anteaya | awesome | 00:25 |
| anteaya | I'm glad you confirmed | 00:25 |
| anteaya | is your gearman worker running? | 00:26 |
| *** jamesmca_ has joined #openstack-infra | 00:26 | |
| *** salv-orlando has joined #openstack-infra | 00:26 | |
| kjorgensen | are you asking if my jenkins slave node shows up in the 'workers'? | 00:27 |
| anteaya | sure | 00:27 |
| anteaya | if you have an answer to that question, let's say that's the question | 00:27 |
| asselin_ | I would like to know if you're using jenkins slaves or jenkins master | 00:28 |
| *** madhu_ak has quit IRC | 00:28 | |
| anteaya | asselin_: thanks for jumping in :) | 00:28 |
| kjorgensen | well our setup has 1 jenkins master and 1 slave node | 00:29 |
| kjorgensen | my test jenkins job is configured to execute only on the slave | 00:29 |
| *** jamesmca_ has quit IRC | 00:29 | |
| asselin_ | ok, btw, there was a bug to run on jenkins master...but should have been fixed & released. Slaves works fine. | 00:29 |
| kjorgensen | so earlier the status looked like this | 00:30 |
| kjorgensen | build:job:SLAVE102 | 00:30 |
| kjorgensen | but now it is like this | 00:30 |
| asselin_ | what's the full output of status? | 00:30 |
| kjorgensen | build:job:SLAVE100 | 00:30 |
| kjorgensen | so the workers seem to have disappeared | 00:30 |
| asselin_ | you should see the job name as part of the output | 00:30 |
| kjorgensen | right, I do see the job name in the full output | 00:31 |
| asselin_ | can you paste it? | 00:31 |
| dims | anteaya : done! | 00:32 |
| anteaya | dims: awesome thanks! | 00:32 |
| *** salv-orlando has quit IRC | 00:32 | |
| kjorgensen | asselin_: our org has many jenkins jobs and they all show up in the output, but I am only interested in the one | 00:33 |
| *** earlephilhower has quit IRC | 00:33 | |
| asselin_ | can you paste the one? | 00:33 |
| openstackgerrit | Merged openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints https://review.openstack.org/301375 | 00:34 |
| kjorgensen | it's what I mentioned before | 00:34 |
| kjorgensen | build:job:SLAVE100 | 00:34 |
| kjorgensen | (I just changed from our company names to generic names) | 00:34 |
| asselin_ | what's the job name and the slave name? | 00:34 |
| asselin_ | i see, job name is 'job', and slave name is 'SLAVE' | 00:35 |
| kjorgensen | yea, basically | 00:35 |
| asselin_ | so that 1 means you have 1 in the queue waiting to run, right? | 00:36 |
| asselin_ | the last 0 means you don't have any slaves capabile of running that job | 00:36 |
| asselin_ | capable* | 00:36 |
| kjorgensen | correct | 00:37 |
| asselin_ | is that slave offline? | 00:37 |
| kjorgensen | no | 00:38 |
| asselin_ | ok, toggle it offline, and bring it backonline and check the status again | 00:38 |
| kjorgensen | we've seen this strange behavior as I've been testing, and to fix it we reset the connection between the jenkins master and the zuul server | 00:39 |
| *** mriedem has quit IRC | 00:39 | |
| asselin_ | i don't recall ever needing to do that... | 00:39 |
| asselin_ | also, do you have the zuul merger running? | 00:40 |
| kjorgensen | when you say "toggle" which service are you referring to? | 00:40 |
| asselin_ | the jenkins slave | 00:40 |
| *** akshai has joined #openstack-infra | 00:40 | |
| asselin_ | in the jenkins master UI, you can take the slave offline and bring it back online | 00:41 |
| *** akshai_ has joined #openstack-infra | 00:42 | |
| kjorgensen | ah, unfortunately I don't have privileges to do that | 00:42 |
| kjorgensen | and yes, zuul-merger is running | 00:42 |
| *** piet has joined #openstack-infra | 00:42 | |
| asselin_ | consider tyring this out and setup your own test environment: http://docs.openstack.org/infra/openstackci/third_party_ci.html | 00:43 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin https://review.openstack.org/312271 | 00:43 |
| openstackgerrit | Madhumitha Viswanathan proposed openstack-infra/jenkins-job-builder: Add partial support for maven-release-plugin https://review.openstack.org/313196 | 00:44 |
| kjorgensen | asselin_: thank you! I will take a look at those instructions | 00:44 |
| kjorgensen | and see if I can apply it to our environment | 00:45 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin https://review.openstack.org/312271 | 00:45 |
| *** akshai has quit IRC | 00:45 | |
| anteaya | at the very least it gives you some freedom to figure out how the tools interact | 00:46 |
| anteaya | so you have a better understanding when you don't have permissions over the entire system | 00:46 |
| *** mriedem has joined #openstack-infra | 00:46 | |
| *** kjorgensen has quit IRC | 00:46 | |
| anteaya | mriedem: I have to say nice work on the recaps | 00:47 |
| anteaya | and it is good to have in print that you are willing to address the nfv issues but lack any interest from anyone willing to do the work | 00:47 |
| anteaya | I'll be pointing folks to that post if it comes up in my earshot | 00:48 |
| *** whoops has quit IRC | 00:48 | |
| *** sflanigan is now known as sflanigan_afk | 00:48 | |
| *** dgonzalez has quit IRC | 00:48 | |
| *** signed8bit_Zzz is now known as signed8bit | 00:49 | |
| *** whoops has joined #openstack-infra | 00:49 | |
| *** frickler has quit IRC | 00:49 | |
| *** tdasilva has quit IRC | 00:50 | |
| *** frickler has joined #openstack-infra | 00:50 | |
| *** dgonzalez has joined #openstack-infra | 00:50 | |
| *** salv-orlando has joined #openstack-infra | 00:51 | |
| *** mtanino has quit IRC | 00:52 | |
| *** tdasilva has joined #openstack-infra | 00:52 | |
| *** bpokorny_ has joined #openstack-infra | 00:53 | |
| *** bpokorny_ has quit IRC | 00:54 | |
| *** bpokorny_ has joined #openstack-infra | 00:55 | |
| *** kjorgensen has joined #openstack-infra | 00:55 | |
| *** salv-orlando has quit IRC | 00:56 | |
| *** bpokorny has quit IRC | 00:57 | |
| *** kjorgens_ has joined #openstack-infra | 00:58 | |
| *** kjorgensen has quit IRC | 00:59 | |
| *** bpokorny_ has quit IRC | 00:59 | |
| *** bpokorny has joined #openstack-infra | 01:00 | |
| *** piet has quit IRC | 01:01 | |
| *** kjorgens_ has quit IRC | 01:02 | |
| *** cody-somerville has joined #openstack-infra | 01:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:03 | |
| *** gyee has quit IRC | 01:03 | |
| *** xarses has joined #openstack-infra | 01:04 | |
| *** kzaitsev_mb has quit IRC | 01:08 | |
| *** gongysh has joined #openstack-infra | 01:09 | |
| *** baoli_ has quit IRC | 01:09 | |
| *** baoli has joined #openstack-infra | 01:10 | |
| *** zhurong has joined #openstack-infra | 01:15 | |
| *** thorst_ has joined #openstack-infra | 01:20 | |
| *** salv-orlando has joined #openstack-infra | 01:22 | |
| *** thorst_ has quit IRC | 01:24 | |
| *** aeng has quit IRC | 01:25 | |
| *** aeng has joined #openstack-infra | 01:26 | |
| mriedem | anteaya: thanks | 01:26 |
| anteaya | :) | 01:27 |
| *** Jeffrey4l has joined #openstack-infra | 01:29 | |
| *** xyang1 has quit IRC | 01:29 | |
| *** salv-orlando has quit IRC | 01:30 | |
| *** chlong has joined #openstack-infra | 01:31 | |
| *** kengo_sakai has joined #openstack-infra | 01:31 | |
| *** ddieterly has quit IRC | 01:31 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Bump u-c oslo.versionedobjects to 1.9.0 https://review.openstack.org/313201 | 01:32 |
| *** yamahata has quit IRC | 01:43 | |
| *** cody-somerville has quit IRC | 01:47 | |
| *** bpokorny has quit IRC | 01:48 | |
| *** Goneri has joined #openstack-infra | 01:48 | |
| *** Swami has quit IRC | 01:49 | |
| *** banix has quit IRC | 01:51 | |
| *** jyuso has quit IRC | 01:52 | |
| *** bpokorny has joined #openstack-infra | 01:54 | |
| *** jcoufal has joined #openstack-infra | 01:57 | |
| *** salv-orlando has joined #openstack-infra | 01:57 | |
| *** baoli has quit IRC | 01:57 | |
| *** camunoz_ has quit IRC | 01:59 | |
| *** bpokorny has quit IRC | 02:00 | |
| *** savihou has quit IRC | 02:00 | |
| *** kjorgensen has joined #openstack-infra | 02:00 | |
| *** savihou has joined #openstack-infra | 02:01 | |
| *** jcoufal has quit IRC | 02:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:04 | |
| *** anteaya has quit IRC | 02:04 | |
| *** kjorgensen has quit IRC | 02:04 | |
| *** sflanigan_afk is now known as sflanigan | 02:04 | |
| *** hdd has quit IRC | 02:05 | |
| *** woodster_ has quit IRC | 02:08 | |
| *** rfolco has quit IRC | 02:08 | |
| *** kzaitsev_mb has quit IRC | 02:09 | |
| *** tonytan4ever has quit IRC | 02:09 | |
| *** salv-orlando has quit IRC | 02:09 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/elastic-recheck: add query for bug 1578866 https://review.openstack.org/313205 | 02:10 |
| openstack | bug 1578866 in OpenStack Identity (keystone) "test_user_update_own_password failing intermittently" [Undecided,New] https://launchpad.net/bugs/1578866 | 02:10 |
| *** Volundr has joined #openstack-infra | 02:10 | |
| *** kjorgensen has joined #openstack-infra | 02:11 | |
| *** whoops has quit IRC | 02:11 | |
| *** shashank_hegde has quit IRC | 02:14 | |
| *** apoorvad has quit IRC | 02:16 | |
| *** kjorgensen has quit IRC | 02:17 | |
| *** piet has joined #openstack-infra | 02:20 | |
| *** vhosakot has quit IRC | 02:25 | |
| *** vhosakot has joined #openstack-infra | 02:31 | |
| *** newbie|2 has joined #openstack-infra | 02:32 | |
| *** vhosakot has quit IRC | 02:33 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add query for bug 1578866 https://review.openstack.org/313205 | 02:35 |
| openstack | bug 1578866 in OpenStack Identity (keystone) "test_user_update_own_password failing intermittently" [High,Confirmed] https://launchpad.net/bugs/1578866 | 02:35 |
| *** mriedem has quit IRC | 02:36 | |
| *** jyuso1 has joined #openstack-infra | 02:36 | |
| *** piet has quit IRC | 02:37 | |
| *** kjorgensen has joined #openstack-infra | 02:38 | |
| *** kjorgensen has quit IRC | 02:42 | |
| *** vhosakot has joined #openstack-infra | 02:47 | |
| *** kjorgensen has joined #openstack-infra | 02:53 | |
| *** piet has joined #openstack-infra | 02:56 | |
| *** ccamacho has quit IRC | 02:57 | |
| *** ccamacho has joined #openstack-infra | 02:58 | |
| *** kjorgensen has quit IRC | 02:58 | |
| *** binbincong has quit IRC | 03:04 | |
| *** daemontool_ has quit IRC | 03:04 | |
| *** kjorgensen has joined #openstack-infra | 03:05 | |
| *** kjorgensen has quit IRC | 03:09 | |
| *** maishsk has quit IRC | 03:10 | |
| *** kjorgensen has joined #openstack-infra | 03:13 | |
| *** piet has quit IRC | 03:13 | |
| *** akshai_ has quit IRC | 03:15 | |
| *** tlian has quit IRC | 03:16 | |
| *** kjorgensen has quit IRC | 03:17 | |
| *** maishsk has joined #openstack-infra | 03:20 | |
| *** markvoelker has quit IRC | 03:20 | |
| *** salv-orlando has joined #openstack-infra | 03:21 | |
| *** hdd has joined #openstack-infra | 03:24 | |
| *** kjorgensen has joined #openstack-infra | 03:28 | |
| *** binbincong has joined #openstack-infra | 03:30 | |
| *** salv-orlando has quit IRC | 03:30 | |
| *** kjorgensen has quit IRC | 03:32 | |
| *** julim has quit IRC | 03:33 | |
| *** amit213 has quit IRC | 03:33 | |
| *** julim has joined #openstack-infra | 03:33 | |
| *** amit213 has joined #openstack-infra | 03:33 | |
| *** signed8bit has quit IRC | 03:36 | |
| *** asettle has quit IRC | 03:42 | |
| *** asettle has joined #openstack-infra | 03:42 | |
| *** salv-orlando has joined #openstack-infra | 03:44 | |
| *** kjorgensen has joined #openstack-infra | 03:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Get translation jobs into healthcheck https://review.openstack.org/311459 | 03:46 |
| *** salv-orlando has quit IRC | 03:46 | |
| *** asettle has quit IRC | 03:47 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Remove the API WG 0000 UTC meeting https://review.openstack.org/313053 | 03:49 |
| *** kjorgensen has quit IRC | 03:50 | |
| *** sflanigan has quit IRC | 03:50 | |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| openstackgerrit | Merged openstack-infra/puppet-kibana: Remove support for Kibana2 https://review.openstack.org/305448 | 03:51 |
| *** esker has quit IRC | 03:55 | |
| *** DevBox has joined #openstack-infra | 03:55 | |
| AJaeger | project-config cores, https://review.openstack.org/313104 is now ready - retiring of two docs repos. Could you review this, please? | 03:55 |
| *** kjorgensen has joined #openstack-infra | 03:57 | |
| *** DevBox has quit IRC | 03:58 | |
| openstackgerrit | Merged openstack-infra/puppet-meetbot: Indentation of the parameters is refactored https://review.openstack.org/295280 | 03:59 |
| *** DevBox has joined #openstack-infra | 04:01 | |
| *** links has joined #openstack-infra | 04:02 | |
| *** kjorgensen has quit IRC | 04:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:05 | |
| *** hamzy has joined #openstack-infra | 04:06 | |
| *** kzaitsev_mb has quit IRC | 04:11 | |
| *** zz_dimtruck is now known as dimtruck | 04:12 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Adding a meeting for Operators Telco and NFV Working Group https://review.openstack.org/312614 | 04:17 |
| *** shashank_hegde has joined #openstack-infra | 04:20 | |
| *** markvoelker has joined #openstack-infra | 04:21 | |
| *** piet has joined #openstack-infra | 04:21 | |
| *** markvoelker has quit IRC | 04:26 | |
| *** salv-orlando has joined #openstack-infra | 04:28 | |
| *** psachin has joined #openstack-infra | 04:30 | |
| *** salv-orlando has quit IRC | 04:32 | |
| *** maishsk has quit IRC | 04:33 | |
| *** Nikolay_St has quit IRC | 04:34 | |
| *** maishsk has joined #openstack-infra | 04:35 | |
| *** hdd has quit IRC | 04:39 | |
| *** maishsk has quit IRC | 04:41 | |
| *** maishsk has joined #openstack-infra | 04:43 | |
| *** dimtruck is now known as zz_dimtruck | 04:46 | |
| *** asettle has joined #openstack-infra | 04:47 | |
| *** cody-somerville has joined #openstack-infra | 04:54 | |
| *** flwang1 has quit IRC | 04:54 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:56 | |
| *** kengo_sa_ has joined #openstack-infra | 05:00 | |
| *** salv-orlando has joined #openstack-infra | 05:01 | |
| *** fedexo has joined #openstack-infra | 05:02 | |
| *** shashank_hegde has quit IRC | 05:04 | |
| *** salv-orlando has quit IRC | 05:04 | |
| *** oanson has joined #openstack-infra | 05:14 | |
| *** salv-orlando has joined #openstack-infra | 05:17 | |
| *** shashank_hegde has joined #openstack-infra | 05:21 | |
| *** markvoelker has joined #openstack-infra | 05:22 | |
| *** salv-orlando has quit IRC | 05:24 | |
| *** kengo_sa_ has quit IRC | 05:25 | |
| *** markvoelker has quit IRC | 05:26 | |
| *** piet has quit IRC | 05:27 | |
| *** netsin has quit IRC | 05:30 | |
| *** netsin has joined #openstack-infra | 05:31 | |
| *** vhosakot has quit IRC | 05:32 | |
| *** Nikolay_St has joined #openstack-infra | 05:32 | |
| *** ianychoi has joined #openstack-infra | 05:32 | |
| *** salv-orlando has joined #openstack-infra | 05:35 | |
| *** jerryz has quit IRC | 05:36 | |
| *** salv-orlando has quit IRC | 05:39 | |
| *** gildub has joined #openstack-infra | 05:44 | |
| *** kzaitsev_mb has quit IRC | 05:52 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Fix validation for stable branch https://review.openstack.org/312350 | 05:53 |
| *** yolanda has joined #openstack-infra | 05:55 | |
| *** mikelk has joined #openstack-infra | 05:56 | |
| *** sridhar_ram has joined #openstack-infra | 05:57 | |
| *** sridhar_ram has quit IRC | 06:04 | |
| *** shashank_hegde has quit IRC | 06:07 | |
| *** shashank_hegde has joined #openstack-infra | 06:14 | |
| *** jkraj has joined #openstack-infra | 06:17 | |
| *** kengo_sa_ has joined #openstack-infra | 06:19 | |
| *** javeriak has joined #openstack-infra | 06:20 | |
| *** zul has quit IRC | 06:21 | |
| *** kengo_sakai has quit IRC | 06:22 | |
| *** markvoelker has joined #openstack-infra | 06:23 | |
| *** itsuugo has quit IRC | 06:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/309273 | 06:25 |
| *** markvoelker has quit IRC | 06:27 | |
| *** asettle has quit IRC | 06:28 | |
| *** fawadkhaliq has joined #openstack-infra | 06:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental-project-config-layout for ubuntu-precise https://review.openstack.org/313182 | 06:32 |
| AJaeger | I'd like an infra-root to cleanup http://tarballs.openstack.org/translation-source/keystone/ and http://tarballs.openstack.org/translation-source/nova/ . Please delete the *.pot files in the directory itself. You can also delete the complete directories, the content is updated after each merge and these files are placed in the wrong location. | 06:34 |
| *** fedexo has quit IRC | 06:36 | |
| *** e0ne has joined #openstack-infra | 06:39 | |
| *** salv-orlando has joined #openstack-infra | 06:46 | |
| *** javeriak has quit IRC | 06:46 | |
| *** yolanda has quit IRC | 06:46 | |
| *** yolanda has joined #openstack-infra | 06:47 | |
| yolanda | AJaeger, done | 06:47 |
| AJaeger | thanks, yolanda ! and good morning! | 06:49 |
| *** yaume has joined #openstack-infra | 06:49 | |
| AJaeger | Now we're ready for https://review.openstack.org/313114 - I'll remove WIP now... | 06:50 |
| *** javeriak has joined #openstack-infra | 06:50 | |
| yolanda | good morning | 06:50 |
| *** yaume has quit IRC | 06:50 | |
| *** yaume has joined #openstack-infra | 06:50 | |
| *** itsuugo has joined #openstack-infra | 06:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Retire operations-guide and ha-guide (2/2) https://review.openstack.org/313105 | 06:53 |
| *** e0ne has quit IRC | 06:53 | |
| AJaeger | yolanda: could you review the project-config reviewing guidelines again, please? https://review.openstack.org/310929 | 06:54 |
| yolanda | sure | 06:55 |
| AJaeger | and is there a meeting going on right now? If not, could you +A https://review.openstack.org/#/c/304213/ , please? | 06:55 |
| *** salv-orlando has quit IRC | 06:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental centos-7 jobs to check-tripleo-jobs https://review.openstack.org/313146 | 06:58 |
| *** e0ne has joined #openstack-infra | 06:59 | |
| *** e0ne has quit IRC | 07:00 | |
| *** yolanda has quit IRC | 07:02 | |
| openstackgerrit | Merged openstack-infra/project-config: stop puppet-unit 3.3 and 3.4 jobs starting from newton https://review.openstack.org/313149 | 07:04 |
| *** permalac has joined #openstack-infra | 07:06 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove statusbot from meeting channels https://review.openstack.org/304213 | 07:06 |
| *** yolanda has joined #openstack-infra | 07:06 | |
| *** yottatsa has joined #openstack-infra | 07:07 | |
| *** shashank_hegde has quit IRC | 07:08 | |
| *** fawadkhaliq has quit IRC | 07:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Set image build in magnum to periodic https://review.openstack.org/312842 | 07:15 |
| openstackgerrit | Merged openstack-infra/project-config: use 24hours everywhere in grafana/neutron.yaml https://review.openstack.org/312950 | 07:16 |
| openstackgerrit | Merged openstack-infra/project-config: Retire operations-guide and ha-guide (1/2) https://review.openstack.org/313104 | 07:16 |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** pahuang has quit IRC | 07:16 | |
| *** ifarkas has joined #openstack-infra | 07:21 | |
| *** yolanda has quit IRC | 07:21 | |
| *** arxcruz has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Set TEMPEST_CONCURRENCY to 1 for grenade-ironic https://review.openstack.org/311225 | 07:22 |
| *** zhurong_ has joined #openstack-infra | 07:23 | |
| *** markvoelker has joined #openstack-infra | 07:23 | |
| *** nadya has joined #openstack-infra | 07:24 | |
| *** oanson has quit IRC | 07:24 | |
| *** zhurong has quit IRC | 07:26 | |
| *** esikachev has joined #openstack-infra | 07:26 | |
| *** salv-orlando has joined #openstack-infra | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-murano-python34-db voting; add gate job https://review.openstack.org/312493 | 07:27 |
| *** markvoelker has quit IRC | 07:28 | |
| *** markus_z has joined #openstack-infra | 07:29 | |
| *** yottatsa has quit IRC | 07:31 | |
| *** zhurong_ has quit IRC | 07:31 | |
| *** yolanda has joined #openstack-infra | 07:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Use m1.large for centos-7 on tripleo-test-cloud-rh1 https://review.openstack.org/312992 | 07:32 |
| *** zhurong has joined #openstack-infra | 07:32 | |
| *** fawadkhaliq has joined #openstack-infra | 07:33 | |
| AJaeger | yolanda: could you put https://review.openstack.org/313114 on your review queue as well, please? Thanks! | 07:35 |
| openstackgerrit | Sylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project https://review.openstack.org/312487 | 07:35 |
| *** oanson has joined #openstack-infra | 07:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Integrate shade+magnum neutron job https://review.openstack.org/312940 | 07:37 |
| *** openstackstatus has quit IRC | 07:38 | |
| *** openstackstatus has joined #openstack-infra | 07:39 | |
| *** ChanServ sets mode: +v openstackstatus | 07:39 | |
| *** hdaniel has joined #openstack-infra | 07:39 | |
| *** cody-somerville has quit IRC | 07:40 | |
| *** salv-orlando has quit IRC | 07:40 | |
| yolanda | AJaeger, sure | 07:40 |
| AJaeger | Thanks, yolanda ! I'd like to wrap up loose ends before my vacation starts in a week... | 07:42 |
| yolanda | oh, vacation ... enjoy :) | 07:43 |
| AJaeger | looking forward to it ;) But first another week... | 07:45 |
| openstackgerrit | Sylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project https://review.openstack.org/312487 | 07:45 |
| *** nadya has quit IRC | 07:46 | |
| openstackgerrit | venkatamahesh proposed openstack/requirements: Remove kolla-mesos from projects.txt https://review.openstack.org/313262 | 07:47 |
| *** jlanoux has joined #openstack-infra | 07:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:47 | |
| *** hdaniel has quit IRC | 07:49 | |
| *** cody-somerville has joined #openstack-infra | 07:52 | |
| *** daemontool has joined #openstack-infra | 07:53 | |
| *** chlong has quit IRC | 07:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Retire operations-guide and ha-guide (2/2) https://review.openstack.org/313105 | 07:55 |
| AJaeger | is the proposal slave offline? I don't see it moving forward any translation jobs... | 07:58 |
| *** zzzeek has quit IRC | 08:00 | |
| *** gildub has quit IRC | 08:01 | |
| *** nadya has joined #openstack-infra | 08:02 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** edwarnicke_ has joined #openstack-infra | 08:04 | |
| *** edwarnicke has quit IRC | 08:07 | |
| *** edwarnicke_ is now known as edwarnicke | 08:07 | |
| *** e0ne has quit IRC | 08:07 | |
| *** andrewbogott_ has joined #openstack-infra | 08:08 | |
| yolanda | AJaeger, checking | 08:08 |
| *** dmk0202 has joined #openstack-infra | 08:08 | |
| *** pilgrimstack1 has joined #openstack-infra | 08:09 | |
| *** pilgrimstack has quit IRC | 08:09 | |
| yolanda | AJaeger, it is | 08:09 |
| *** andrewbogott has quit IRC | 08:09 | |
| *** andrewbogott_ is now known as andrewbogott | 08:09 | |
| yolanda | there may be some job stopping it | 08:09 |
| *** BobBall_AWOL is now known as BobBall | 08:11 | |
| *** lucas-afk is now known as lucasagomes | 08:15 | |
| *** salv-orlando has joined #openstack-infra | 08:19 | |
| yolanda | AJaeger, i'm checking the latest jobs now | 08:19 |
| *** e0ne has joined #openstack-infra | 08:20 | |
| yolanda | looks as the puppet constraints job set proposal offline | 08:21 |
| *** kushal has joined #openstack-infra | 08:23 | |
| *** mhickey has joined #openstack-infra | 08:23 | |
| AJaeger | argh ;( | 08:23 |
| AJaeger | ok, will check and sent a patch... | 08:23 |
| AJaeger | thanks , yolanda | 08:24 |
| *** salv-orlando has quit IRC | 08:24 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Setup propose-puppet-openstack-constraints properly https://review.openstack.org/313271 | 08:27 |
| AJaeger | yolanda: is that regex fine? ^ | 08:27 |
| *** sflanigan has quit IRC | 08:27 | |
| *** shardy has joined #openstack-infra | 08:28 | |
| yolanda | AJaeger, going to double check... | 08:28 |
| AJaeger | please do ;) | 08:28 |
| yolanda | looks good | 08:29 |
| *** jaosorior has joined #openstack-infra | 08:32 | |
| *** jistr has joined #openstack-infra | 08:32 | |
| *** jyuso1 has quit IRC | 08:33 | |
| *** javeriak has quit IRC | 08:33 | |
| *** jyuso1 has joined #openstack-infra | 08:34 | |
| *** pcaruana has joined #openstack-infra | 08:36 | |
| *** dizquierdo has joined #openstack-infra | 08:37 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** e0ne has quit IRC | 08:40 | |
| *** jang has joined #openstack-infra | 08:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Setup propose-puppet-openstack-constraints properly https://review.openstack.org/313271 | 08:40 |
| *** gongysh has quit IRC | 08:40 | |
| *** rossella_s has joined #openstack-infra | 08:41 | |
| *** binbincong has quit IRC | 08:41 | |
| *** binbincong has joined #openstack-infra | 08:42 | |
| *** e0ne has joined #openstack-infra | 08:42 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Fix validation for stable branch https://review.openstack.org/312350 | 08:44 |
| *** salv-orlando has joined #openstack-infra | 08:44 | |
| *** gongysh has joined #openstack-infra | 08:44 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add VirtualBMC to gerritbot https://review.openstack.org/313280 | 08:46 |
| *** salv-orlando has quit IRC | 08:48 | |
| *** zeih has joined #openstack-infra | 08:53 | |
| *** rossella_s has quit IRC | 08:54 | |
| *** fawadkhaliq has quit IRC | 08:54 | |
| *** rossella_s has joined #openstack-infra | 08:54 | |
| *** ihrachys has joined #openstack-infra | 08:54 | |
| *** jordanP has joined #openstack-infra | 08:55 | |
| *** tesseract has joined #openstack-infra | 08:56 | |
| *** tesseract is now known as Guest42930 | 08:56 | |
| *** gongysh has quit IRC | 08:59 | |
| *** alex_xu has quit IRC | 09:03 | |
| *** nadya has quit IRC | 09:04 | |
| *** alex_xu has joined #openstack-infra | 09:05 | |
| *** javeriak has joined #openstack-infra | 09:06 | |
| *** electrofelix has joined #openstack-infra | 09:07 | |
| bunting | lifeless: I was wondering if upper constraints worked on the clients? | 09:11 |
| *** cody-somerville has quit IRC | 09:12 | |
| *** zeih has quit IRC | 09:12 | |
| *** sambetts|afk is now known as sambetts | 09:14 | |
| *** javeriak has quit IRC | 09:20 | |
| *** dizquierdo has quit IRC | 09:30 | |
| *** javeriak has joined #openstack-infra | 09:31 | |
| *** chem has joined #openstack-infra | 09:33 | |
| *** asettle has joined #openstack-infra | 09:35 | |
| *** yottatsa has joined #openstack-infra | 09:36 | |
| *** jyuso1 has quit IRC | 09:38 | |
| *** asettle has quit IRC | 09:39 | |
| *** salv-orlando has joined #openstack-infra | 09:41 | |
| *** maishsk has quit IRC | 09:44 | |
| *** maishsk_ has joined #openstack-infra | 09:44 | |
| *** zhurong has quit IRC | 09:46 | |
| *** mhickey has quit IRC | 09:47 | |
| *** tosky has joined #openstack-infra | 09:47 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/system-config: Refresh log files for tripleo project https://review.openstack.org/312985 | 09:48 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Move ansible-dev magnum shade to non-voting https://review.openstack.org/313301 | 09:49 |
| *** salv-orlando has quit IRC | 09:49 | |
| *** gildub has joined #openstack-infra | 09:49 | |
| *** zhurong has joined #openstack-infra | 09:51 | |
| *** pfallenop has quit IRC | 09:53 | |
| *** jkraj has quit IRC | 09:57 | |
| *** sshnaidm has quit IRC | 09:57 | |
| *** jaosorior is now known as jaosorior_lunch | 09:59 | |
| *** pfallenop has joined #openstack-infra | 10:00 | |
| *** mhickey has joined #openstack-infra | 10:02 | |
| *** ihrachys has quit IRC | 10:04 | |
| *** zhurong has quit IRC | 10:06 | |
| *** maishsk_ has quit IRC | 10:08 | |
| *** ekarlso has quit IRC | 10:08 | |
| *** zhurong has joined #openstack-infra | 10:08 | |
| kozhukalov | team, could you please merge this https://review.openstack.org/#/c/298615/ ? | 10:10 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Add gate-tempest-dsvm-neutron-full-ssh to periodic https://review.openstack.org/313051 | 10:10 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add bay certificates to shade https://review.openstack.org/313308 | 10:10 |
| *** zhurong has quit IRC | 10:11 | |
| Zara | Hi, I'm in the process of patching http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/users.pp to add my credentials; does realname map to a name used anywhere else in the system? I'm asking as I'm often just 'Zara' sans surname. | 10:12 |
| nibalizer | Zara: totally fine | 10:13 |
| *** zhurong has joined #openstack-infra | 10:13 | |
| *** zhurong has quit IRC | 10:13 | |
| rcarrillocruz | it's too early for you nibalizer , where are thou? | 10:14 |
| yolanda | kozhukalov, approved. The depends-on already landed so is fine now | 10:14 |
| nibalizer | rcarrillocruz: :) | 10:14 |
| nibalizer | still in portland | 10:14 |
| Zara | nibalizer: so it's okay for me to use my surname or not as I wish? :) | 10:14 |
| nibalizer | yah | 10:14 |
| Zara | \o/ | 10:14 |
| rcarrillocruz | btw nibalizer | 10:16 |
| rcarrillocruz | https://github.com/ansible/ansible-modules-extras/pull/2173 | 10:16 |
| rcarrillocruz | initial designate ansible goodness coming | 10:16 |
| *** electrofelix has quit IRC | 10:20 | |
| *** dizquierdo has joined #openstack-infra | 10:20 | |
| *** dims has quit IRC | 10:20 | |
| *** chuck__ has joined #openstack-infra | 10:25 | |
| *** markvoelker has joined #openstack-infra | 10:26 | |
| *** sdague has joined #openstack-infra | 10:28 | |
| *** dizquierdo has quit IRC | 10:29 | |
| *** markvoelker has quit IRC | 10:30 | |
| *** electrofelix has joined #openstack-infra | 10:32 | |
| *** kjorgensen has joined #openstack-infra | 10:33 | |
| *** alaski is now known as lascii | 10:36 | |
| *** maishsk has joined #openstack-infra | 10:36 | |
| *** kjorgensen has quit IRC | 10:37 | |
| *** dizquierdo has joined #openstack-infra | 10:44 | |
| *** maishsk has quit IRC | 10:46 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Decouple Shotgun project from Fuel https://review.openstack.org/298615 | 10:48 |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Decouple Shotgun project from Fuel https://review.openstack.org/298615 | 10:49 |
| *** pavel_bondar has joined #openstack-infra | 10:51 | |
| *** Goneri has quit IRC | 10:52 | |
| *** nadya has joined #openstack-infra | 10:55 | |
| *** dmk0202 has quit IRC | 10:57 | |
| *** dmk0202 has joined #openstack-infra | 10:57 | |
| *** pavel_bondar has quit IRC | 10:57 | |
| *** oanson has quit IRC | 10:57 | |
| *** pavel_bondar has joined #openstack-infra | 10:58 | |
| *** Hal has joined #openstack-infra | 10:58 | |
| *** Hal is now known as Guest98690 | 10:59 | |
| *** asettle has joined #openstack-infra | 10:59 | |
| *** asettle has quit IRC | 10:59 | |
| *** jtomasek has quit IRC | 10:59 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add bay certificates to shade https://review.openstack.org/313308 | 11:00 |
| *** kushal has quit IRC | 11:00 | |
| *** dims has joined #openstack-infra | 11:02 | |
| *** maishsk has joined #openstack-infra | 11:03 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/requirements: [sahara] remove sahara-tests from projects.txt https://review.openstack.org/313501 | 11:04 |
| *** chuck__ has quit IRC | 11:06 | |
| *** mhickey has quit IRC | 11:07 | |
| *** esikachev has quit IRC | 11:07 | |
| *** yamamoto has quit IRC | 11:08 | |
| *** oanson has joined #openstack-infra | 11:10 | |
| sylwesterB | Folks, could you check it out? https://review.openstack.org/#/c/312487/ if everything is done? | 11:11 |
| *** jaosorior_lunch has quit IRC | 11:11 | |
| *** jaosorior_lunch has joined #openstack-infra | 11:12 | |
| *** jaosorior_lunch is now known as jaosorior | 11:12 | |
| *** javeriak has quit IRC | 11:19 | |
| *** mhickey has joined #openstack-infra | 11:20 | |
| *** Na3iL has joined #openstack-infra | 11:20 | |
| *** maishsk has quit IRC | 11:21 | |
| *** maishsk has joined #openstack-infra | 11:24 | |
| *** dizquierdo has quit IRC | 11:25 | |
| ansiwen | AJaeger: how long does it take until the requirements land in nove? any idea? | 11:25 |
| *** esikachev has joined #openstack-infra | 11:26 | |
| *** markvoelker has joined #openstack-infra | 11:27 | |
| openstackgerrit | Zara proposed openstack-infra/system-config: Add Zara to users https://review.openstack.org/313510 | 11:27 |
| Zara | well, that took ages; got distracted by email | 11:27 |
| *** kushal has joined #openstack-infra | 11:27 | |
| AJaeger | ansiwen: let's ask dhellmann or sdague to review 311761 as first step | 11:27 |
| *** chuck__ has joined #openstack-infra | 11:28 | |
| *** kengo_sakai has joined #openstack-infra | 11:28 | |
| *** javeriak has joined #openstack-infra | 11:28 | |
| *** ccamacho is now known as ccamacho|lunch | 11:30 | |
| sdague | AJaeger: +A on the requirements review | 11:31 |
| *** markvoelker has quit IRC | 11:31 | |
| openstackgerrit | Jianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool https://review.openstack.org/310698 | 11:33 |
| openstackgerrit | Jianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool https://review.openstack.org/310698 | 11:35 |
| *** kengo_sakai has quit IRC | 11:35 | |
| AJaeger | thanks, sdague ! ansiwen, once that one is in, a change against nova will be proposed and than one needs to merge.. - and then https://review.openstack.org/311753 needs rebasing and can merge... | 11:36 |
| openstackgerrit | Jianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool https://review.openstack.org/310698 | 11:37 |
| sdague | AJaeger: 311753 can also change tox.ini as well, right? | 11:37 |
| AJaeger | sdague: what should be changed? | 11:38 |
| AJaeger | We use the venv environment for the translations | 11:38 |
| AJaeger | sdague: speaking about translations: https://review.openstack.org/313114 does not store .pot files anymore in repos, if you want to review it... | 11:39 |
| sdague | AJaeger: https://github.com/openstack/nova/blob/449c76f0ee67b43ce6ff8a76546aba7328250694/tox.ini#L30-L32 | 11:39 |
| AJaeger | yeah, that could be removed - it's done by infra scripts now | 11:39 |
| AJaeger | ansiwen: ^ | 11:40 |
| ansiwen | AJaeger: btw, how are the po files generated from the pot files? | 11:40 |
| *** tbachman has left #openstack-infra | 11:40 | |
| sdague | AJaeger: right, but we put that into the tox.ini so that we'd get end user fail | 11:40 |
| *** alex_xu has quit IRC | 11:42 | |
| *** jang has quit IRC | 11:42 | |
| AJaeger | ansiwen: our scripts generate everything and push to repos - there shouldn't be a need to have pot files in there. | 11:42 |
| *** baoli has joined #openstack-infra | 11:42 | |
| *** dizquierdo has joined #openstack-infra | 11:42 | |
| AJaeger | ansiwen: po files are done on translation server | 11:42 |
| *** DevBox has quit IRC | 11:42 | |
| *** tlian has joined #openstack-infra | 11:42 | |
| AJaeger | sdague: we only recently put those tox.ini lines into the infra scripts | 11:43 |
| ansiwen | AJaeger: that was my question... since setup.py/babel also doesn't support several domains for updateing po files from the pot files, I assumed the po files are generated elsewhere | 11:43 |
| *** yolanda has quit IRC | 11:43 | |
| AJaeger | ansiwen: we generate pot files in CI, push to zanata after each merge - and once a day check for translated pot files in zanata and download them and propose to projects, see https://review.openstack.org/#/q/status:open+topic:zanata/translations,n,z for current proposals | 11:44 |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| ansiwen | ok, I see... | 11:45 |
| *** penguinolog has quit IRC | 11:46 | |
| *** maishsk has quit IRC | 11:46 | |
| ansiwen | AJaeger: because I realized I'm not very happy with my babel change, because it's not consistent. compile_catalogs (po->mo) supports several domains now, but init_catalogs and update_catalogs (pot->po) doesn't support several domains. feels a bit ugly. | 11:46 |
| *** zhurong has joined #openstack-infra | 11:47 | |
| AJaeger | projects don't need init_catalogs and update_catalogs in the OpenStack CI environment | 11:47 |
| ansiwen | for openstack it doesn't matter, because update_catalogs is not used | 11:47 |
| *** alex_xu has joined #openstack-infra | 11:47 | |
| amrith | is something the matter with zuul? the web UI appears to be unresponsive (http://zuul.openstack.org) | 11:47 |
| *** baoli_ has joined #openstack-infra | 11:47 | |
| AJaeger | ansiwen: exactly. But if it's used outside, it's indeed different. | 11:47 |
| ansiwen | AJaeger: right, but it's ugly in general | 11:48 |
| *** yolanda has joined #openstack-infra | 11:48 | |
| AJaeger | agreed | 11:48 |
| *** baoli has quit IRC | 11:50 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Amend the valid fields to update on recordsets https://review.openstack.org/313520 | 11:51 |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** salv-orlando has joined #openstack-infra | 11:58 | |
| *** psilvad has joined #openstack-infra | 11:58 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** yamamoto has joined #openstack-infra | 12:05 | |
| *** rfolco has joined #openstack-infra | 12:07 | |
| *** psilvad has quit IRC | 12:07 | |
| *** asettle has joined #openstack-infra | 12:07 | |
| *** rhallisey has joined #openstack-infra | 12:07 | |
| *** aysyd has joined #openstack-infra | 12:09 | |
| *** lucasagomes is now known as lucas-hungry | 12:11 | |
| *** dprince has joined #openstack-infra | 12:11 | |
| *** dingyichen has quit IRC | 12:12 | |
| pabelanger | morning | 12:12 |
| *** markvoelker has joined #openstack-infra | 12:14 | |
| *** maestro has joined #openstack-infra | 12:16 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** Jeffrey4l has quit IRC | 12:17 | |
| *** trown|outtypewww is now known as trown | 12:17 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** psilvad has joined #openstack-infra | 12:18 | |
| *** ekarlso has joined #openstack-infra | 12:18 | |
| *** kgiusti has joined #openstack-infra | 12:19 | |
| *** ddieterly has joined #openstack-infra | 12:19 | |
| *** yamamoto has quit IRC | 12:19 | |
| *** yamamoto has joined #openstack-infra | 12:20 | |
| *** ddieterly is now known as ddieterly[away] | 12:20 | |
| AJaeger | morning, pabelanger ! | 12:20 |
| *** gordc has joined #openstack-infra | 12:21 | |
| openstackgerrit | Anant Patil proposed openstack-infra/project-config: Add experimental magnum job to heat https://review.openstack.org/313525 | 12:21 |
| *** alex_xu has quit IRC | 12:22 | |
| *** binbincong has quit IRC | 12:23 | |
| *** binbincong has joined #openstack-infra | 12:24 | |
| *** alex_xu has joined #openstack-infra | 12:25 | |
| openstackgerrit | Kiall Mac Innes proposed openstack-infra/project-config: Add experimental job for Designate's knot2 agent backend https://review.openstack.org/313530 | 12:27 |
| pabelanger | So, I think somebody patch bombed review.o.o | 12:27 |
| pabelanger | openstack/monasca-analytics seems to have uploaded 175 patches, in a row | 12:27 |
| AJaeger | Wow! | 12:28 |
| *** salv-orlando has quit IRC | 12:28 | |
| *** oanson has quit IRC | 12:28 | |
| *** Jeffrey4l has joined #openstack-infra | 12:28 | |
| AJaeger | I just checked status.o.o/zuul and was surprised to see a long stack at the top | 12:28 |
| pabelanger | Ya, zuul is doing its best to handle it | 12:28 |
| pabelanger | but status.o.o/zuul is slow | 12:29 |
| AJaeger | https://review.openstack.org/#/c/313346/ - 177 related changes. 177! | 12:29 |
| pabelanger | yup | 12:29 |
| *** amrith is now known as _amrith_ | 12:29 | |
| *** rodrigods has quit IRC | 12:29 | |
| *** rodrigods has joined #openstack-infra | 12:29 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Add gate-tempest-dsvm-neutron-full-ssh to periodic https://review.openstack.org/313051 | 12:29 |
| AJaeger | that series looks really wrong, looking at titles | 12:29 |
| *** yamamoto has quit IRC | 12:29 | |
| AJaeger | "Fix test and addressed comments." or "Fix pep8 new checks." | 12:30 |
| *** psilvad has quit IRC | 12:30 | |
| AJaeger | A lot should be squeezed together... | 12:30 |
| StevenK | win 22 | 12:30 |
| pabelanger | david.perez5@hpe.com is the committer | 12:31 |
| AJaeger | who will send him a link to the Developer's Guide? ;) | 12:31 |
| * AJaeger has coffee break now... | 12:31 | |
| *** psilvad has joined #openstack-infra | 12:31 | |
| *** regXboi has joined #openstack-infra | 12:32 | |
| *** regXboi has quit IRC | 12:33 | |
| pabelanger | Ya, zuul is trying to noop them all | 12:33 |
| pabelanger | which is just taking time | 12:33 |
| *** regXboi has joined #openstack-infra | 12:33 | |
| pabelanger | that explains the lack of movement on nodepool.o.o too | 12:33 |
| *** kushal has quit IRC | 12:34 | |
| *** yamamoto has joined #openstack-infra | 12:35 | |
| *** kengo_sa_ has quit IRC | 12:35 | |
| *** lhinds has joined #openstack-infra | 12:35 | |
| *** signed8bit has joined #openstack-infra | 12:36 | |
| *** kengo_sakai has joined #openstack-infra | 12:36 | |
| lhinds | hello all, where would be the right place to report an issue with the wiki - launchpad OpenID integration? | 12:36 |
| lhinds | I keep getting 'Verification of the OpenID URL failed. Error message: "Nonce already used or out of range"' | 12:37 |
| lhinds | yet it works fine for gerrit | 12:37 |
| pabelanger | lhinds: you are in the correct spot. | 12:38 |
| *** xyang1 has joined #openstack-infra | 12:38 | |
| pabelanger | lhinds: what is your wiki.o.o username? | 12:38 |
| lhinds | lhinds: wiki is lhinds | 12:38 |
| lhinds | https://launchpad.net/~lhinds | 12:39 |
| *** savihou has quit IRC | 12:39 | |
| pabelanger | lhinds: There is no user by the name "lhinds". Check your spelling. | 12:40 |
| *** Guest98690 has quit IRC | 12:40 | |
| pabelanger | have you logged into the wiki before? | 12:40 |
| lhinds | pabelanger, good question, I don't think I have | 12:40 |
| lhinds | been trying via launchpad | 12:40 |
| pabelanger | ya, so that is the issue. We've disabled new user creation on wiki.o.o | 12:40 |
| pabelanger | because of spamming issues | 12:40 |
| pabelanger | let me see if I can manually create your account | 12:41 |
| lhinds | thanks pabelanger | 12:41 |
| *** kushal has joined #openstack-infra | 12:41 | |
| lhinds | Full name: Luke Hinds | 12:41 |
| lhinds | Username: lhinds | 12:41 |
| lhinds | Email address: lhinds@redhat.com | 12:41 |
| *** kengo_sakai has quit IRC | 12:41 | |
| *** mat128 is now known as mat128|afk | 12:42 | |
| *** Goneri has joined #openstack-infra | 12:44 | |
| *** Na3iL has quit IRC | 12:46 | |
| *** ddieterly[away] is now known as ddieterly | 12:46 | |
| Nikolay_St | AJaeger: fungi can you help me with fail of this patch: https://review.openstack.org/#/c/310138/ ? fuel-web guys said that it can be infrastructure issue | 12:47 |
| *** tpsilva has joined #openstack-infra | 12:47 | |
| *** javeriak has quit IRC | 12:47 | |
| pabelanger | lhinds: Hmm, not sure I can manually create it. Let me talk with fungi to see what we need to do. But keep in mind, we are actually deprecating the wiki so depending on the content, there may be a better place to place it today | 12:48 |
| *** yamamoto has quit IRC | 12:48 | |
| *** ayoung has quit IRC | 12:48 | |
| lhinds | thanks pabelanger - I need to put up a security note track number, I can see if someone else can make the edit for me. | 12:48 |
| *** asettle has quit IRC | 12:49 | |
| lhinds | please do check with fungi as well still | 12:49 |
| *** |-paul-| has joined #openstack-infra | 12:49 | |
| pabelanger | lhinds: okay, ya. It is not the ideal thing, but likely solves the current issue. We'll ping you once we figure it out | 12:49 |
| lhinds | thanks! | 12:49 |
| AJaeger | Nikolay_St: I see a failing request - no idea where that comes from. You need to dig deeper and debug this yourself | 12:49 |
| *** Na3iL has joined #openstack-infra | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:50 | |
| *** gildub has quit IRC | 12:50 | |
| AJaeger | pabelanger: did you email the monasca person? Otherwise I'll do it now... | 12:50 |
| pabelanger | AJaeger: fungi: So, I wonder if we need to dump the check queue for openstack/monasca-analytics or just let zuul continue to process the changes. At this rate is it going to be a few hours before zuul clears out | 12:51 |
| pabelanger | AJaeger: I have not | 12:51 |
| pabelanger | AJaeger: I was actually looking for their IRC handel | 12:51 |
| pabelanger | handle* | 12:51 |
| *** hrybacki|gone is now known as hrybacki | 12:51 | |
| AJaeger | It's #openstack-monasca | 12:52 |
| pabelanger | okay, heading there | 12:52 |
| AJaeger | Good idea, let's ask there - on my way... | 12:52 |
| *** ccamacho|lunch is now known as ccamacho | 12:52 | |
| *** akshai has joined #openstack-infra | 12:53 | |
| *** mat128|afk is now known as mat128 | 12:55 | |
| pabelanger | AJaeger: fungi: Ah, I keep forgetting, the only way to dump a queue is to restart zuul | 12:55 |
| *** jheroux has joined #openstack-infra | 12:55 | |
| *** ddieterly has quit IRC | 12:56 | |
| *** mhickey has quit IRC | 12:57 | |
| *** thiagop-afk has joined #openstack-infra | 12:58 | |
| *** thiagop-afk is now known as thiagop | 12:58 | |
| *** sdake has joined #openstack-infra | 13:00 | |
| *** ifarkas has quit IRC | 13:01 | |
| *** nadya has quit IRC | 13:01 | |
| *** jaypipes has joined #openstack-infra | 13:02 | |
| AJaeger | pabelanger: back to normal business ;) | 13:02 |
| AJaeger | pabelanger: seems that other channel is just idle ;( | 13:02 |
| *** yolanda has quit IRC | 13:02 | |
| *** matt-borland has joined #openstack-infra | 13:02 | |
| pabelanger | possible | 13:02 |
| *** yottatsa has quit IRC | 13:02 | |
| *** amitgandhinz has joined #openstack-infra | 13:04 | |
| *** davidSubirosPere has joined #openstack-infra | 13:04 | |
| AJaeger | Let's see whether we get any reaction... | 13:05 |
| *** anteaya has joined #openstack-infra | 13:06 | |
| davidSubirosPere | hi, I'm David, I was asked to join this IRC channel to talk about some code I pushed that contains too many commits | 13:06 |
| *** psachin has quit IRC | 13:06 | |
| *** piet has joined #openstack-infra | 13:06 | |
| anteaya | hi davidSubirosPere can you share the url of the gerrit change? | 13:06 |
| davidSubirosPere | yes | 13:07 |
| davidSubirosPere | https://review.openstack.org/#/c/313354 | 13:07 |
| *** yolanda has joined #openstack-infra | 13:07 | |
| davidSubirosPere | I work in Hewlett Packard Enterprise, and we are releasing some code as open source, I wanted to keep the git history, but it seems there are too many commits | 13:08 |
| *** yottatsa has joined #openstack-infra | 13:08 | |
| anteaya | pabelanger: ^^ regarding your email | 13:08 |
| AJaeger | davidSubirosPere: The proper way to keep history is to import it at repo setup time, not afterwards. | 13:09 |
| AJaeger | Now somebody has to manually merge 178 changes via the webui ;( | 13:09 |
| AJaeger | davidSubirosPere: thanks for showing up! | 13:09 |
| mescanef | hello infra team:-) please review some puppet code: https://review.openstack.org/#/c/299661/ and https://review.openstack.org/#/c/303306/ | 13:11 |
| *** jsavak has joined #openstack-infra | 13:12 | |
| *** yamamoto has quit IRC | 13:13 | |
| AJaeger | davidSubirosPere: by accident you did a denial-of-service attack on Zuul - the many changes with the long dependencies of these changes keep our CI really busy... | 13:13 |
| *** |-paul-| has quit IRC | 13:13 | |
| EmilienM | pabelanger: o/ | 13:14 |
| davidSubirosPere | sorry about that :( is there a way to stop the jobs? | 13:14 |
| AJaeger | davidSubirosPere: Not that I'm aware of - just don't resend the whole series, please! | 13:14 |
| *** savihou has joined #openstack-infra | 13:15 | |
| AJaeger | we currently have 11050 changes in the check queue - new record ;) http://grafana.openstack.org/dashboard/db/zuul-status | 13:15 |
| dhellmann | AJaeger : if he abandons the first patch in the series would that kill all of the jobs behind it? and then we could import the history some other way? | 13:15 |
| davidSubirosPere | ok, I thought only a single job would be generated, from now on I will not send so many commits :) | 13:15 |
| *** yamamoto has joined #openstack-infra | 13:15 | |
| AJaeger | dhellmann: no idea | 13:16 |
| Kiall | dhellmann: updating the first one on the chain will cause the others to be aborted | 13:16 |
| Kiall | even a commit message update would do it | 13:16 |
| dhellmann | Kiall : and they wouldn't just be restarted? | 13:16 |
| AJaeger | davidSubirosPere: We test each patch on its own. So, if you submit two changes, we test the first one on its own, then the second with the first one applied... | 13:16 |
| Kiall | dhellmann: Correct, they won't be restarted | 13:16 |
| AJaeger | Kiall: if you edit the commit message, you restart. Abort might work - no idea | 13:16 |
| Kiall | (not without a recheck) | 13:16 |
| Kiall | Well - the first patch will restart | 13:17 |
| Kiall | but the 176 dependant's will not | 13:17 |
| *** mtanino has joined #openstack-infra | 13:17 | |
| AJaeger | Kiall: yeah, you might be right since they now would all be not dependent - so merge conflcits basically | 13:17 |
| dhellmann | cool, so that might be a thing to do -- edit the message using the gerrit ui, then wait for the other jobs to die and abandon those patches | 13:17 |
| AJaeger | fungi, are you around to help? | 13:17 |
| davidSubirosPere | shall I abandon the change? | 13:17 |
| dhellmann | davidSubirosPere : don't do anything yet, let's make sure we have consensus on what will happen :-) | 13:18 |
| *** yamamoto has quit IRC | 13:18 | |
| davidSubirosPere | ok | 13:18 |
| AJaeger | davidSubirosPere: ++ | 13:18 |
| *** lucas-hungry is now known as lucasagomes | 13:18 | |
| pabelanger | anteaya: davidSubirosPere: Thanks for joining | 13:18 |
| AJaeger | I agree with dhellmann, let's spend some time on figuring out best solution | 13:18 |
| *** piet has quit IRC | 13:19 | |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| anteaya | I'm not buying changing the first commit message aborts jobs on the other patches | 13:19 |
| pabelanger | if we could go back in time, we could just import the project with jeepby but not sure that will work | 13:19 |
| pabelanger | couldn't a infra-root do a force push at this point? | 13:19 |
| *** piet has joined #openstack-infra | 13:20 | |
| dhellmann | first priority should be unclogging the gate, then we can figure out the import | 13:20 |
| dhellmann | unless you think a force push would somehow unclog the gate? | 13:20 |
| *** mriedem has joined #openstack-infra | 13:20 | |
| anteaya | I only see 32 patches in the gate | 13:20 |
| dhellmann | anteaya : sorry, "check queue" | 13:21 |
| *** yamamoto has quit IRC | 13:21 | |
| anteaya | okay with you now | 13:21 |
| AJaeger | anteaya: we speak about check queue - 10k jobs | 13:21 |
| anteaya | okay | 13:21 |
| sdague | AJaeger: but they are all noop | 13:21 |
| anteaya | has zuul stopped working? | 13:21 |
| AJaeger | I mean: 10k changes... | 13:21 |
| pabelanger | I think Kiall is right, if we update the first patch in the stack, zuul should know that and kill off the other changes in the check queue | 13:21 |
| anteaya | pabelanger: I'm not convincing | 13:21 |
| anteaya | convinced | 13:21 |
| AJaeger | sdague: Yes, but Zuul still needs to process them and that seems to take exponential time ;( | 13:21 |
| dhellmann | pabelanger : the question is whether they die, or get restarted | 13:22 |
| anteaya | I think they will all get restarted | 13:22 |
| sdague | pabelanger: I'm not super convinced of that, they may just get restarted | 13:22 |
| dhellmann | maybe someone can write an abandon script? | 13:22 |
| * AJaeger doesn't know - and Zuul is too busy to test anything ;/ | 13:22 | |
| pabelanger | dhellmann: sdague: right, so we need a merge conflict between patch 1 or patch 2. | 13:22 |
| sdague | what happens if these get approved | 13:22 |
| Kiall | anteaya: I'm 99.9% sure they will not be restarted. | 13:22 |
| Kiall | Upload 2 dependant changes somewhere and try it.. | 13:22 |
| sdague | Kiall: you can't try anything right now | 13:23 |
| pabelanger | we can also dump both check and gate queue, restart zuul, then load gate again | 13:23 |
| *** yamamoto has joined #openstack-infra | 13:23 | |
| sdague | because it's all jammed up | 13:23 |
| pabelanger | asking people in check to recheck their patches | 13:23 |
| jroll | change the first patch to be 'git rm -rf *' ? that'll merge conflict for sure | 13:23 |
| anteaya | pabelanger: yeah I think that is the way to go | 13:23 |
| Kiall | I can confirm that behaviour on our internal Gerrit/Zuul? | 13:23 |
| anteaya | save the gate queue and clear the check queue | 13:23 |
| *** banix has joined #openstack-infra | 13:24 | |
| anteaya | davidSubirosPere: and don't do anything with your patch series until we discuss a plan please | 13:24 |
| sdague | zuul is definitely processing these though, I do kind of wonder if it's better to just script approve them | 13:24 |
| anteaya | I'm only seeing 250 in check btw | 13:24 |
| sdague | it will take some time to chug through it | 13:24 |
| anteaya | according to status.openstack.org/zuul | 13:24 |
| *** rhochmuth has joined #openstack-infra | 13:24 | |
| anteaya | graphana counts all the dependencies | 13:24 |
| pabelanger | sdague: right, at this point it is just a waiting game | 13:24 |
| *** pcaruana has quit IRC | 13:25 | |
| anteaya | so 10k in the check queue in grafana doesn't mean 10K patches are getting tested | 13:25 |
| AJaeger | anteaya: Zuul needs to handle those 10k internally. | 13:25 |
| sdague | right, but if there is an exponential effect then as things merge, they speed up | 13:25 |
| sdague | because zuul doesn't have to process all the chain | 13:25 |
| anteaya | right so that was my early question, has zuul stopped? | 13:25 |
| AJaeger | anteaya: Yes, those are 178 changes stacked on each other - plus those submitted by others - that's why we have 250 now | 13:25 |
| anteaya | is zuul processing events or has it ceased? | 13:25 |
| pabelanger | anteaya: no, zuul is still working, just slowly | 13:25 |
| sdague | no infra-root around? | 13:25 |
| anteaya | pabelanger: great thank you | 13:26 |
| pabelanger | I am | 13:26 |
| sdague | ah, ok | 13:26 |
| anteaya | sdague: pabelanger is root | 13:26 |
| sdague | cool | 13:26 |
| sdague | so, I would see about approving the first 10 patches in the stack, and see how long they take to get through | 13:26 |
| dhellmann | so does someone have a script for approving lots of patches? :-) | 13:26 |
| sdague | gate queue has priority | 13:26 |
| *** jcoufal has joined #openstack-infra | 13:26 | |
| sdague | if that moves ok, then that's probably a good approach | 13:26 |
| anteaya | why have we moved to approving? | 13:27 |
| yolanda | hi, i am around as well, but my connection is so bad | 13:27 |
| anteaya | I'm not following that logic? | 13:27 |
| *** yamamoto has quit IRC | 13:27 | |
| anteaya | the world is not on fire, zuul is just slow right now | 13:27 |
| AJaeger | the PTL should approve! davidSubirosPere is not the PTL for that... | 13:27 |
| anteaya | we are not in a release week | 13:27 |
| *** yamamoto has joined #openstack-infra | 13:27 | |
| jroll | worth noting is that this repo has only noop-jobs configured | 13:27 |
| anteaya | I'm inclined to let zuul do it's job | 13:27 |
| sdague | jroll: right, so it's only zuul provessing time | 13:27 |
| *** mtanino has quit IRC | 13:27 | |
| jroll | ya, no node waste | 13:27 |
| sdague | anteaya: right, that's what I'm saying | 13:27 |
| rhochmuth | Ajaeger: I was just expecting one commit. Is there a way to cancel all those reviews? | 13:27 |
| *** ewilson has joined #openstack-infra | 13:28 | |
| rhochmuth | for monasca-analytics | 13:28 |
| *** ddieterly has joined #openstack-infra | 13:28 | |
| *** ddieterly is now known as ddieterly[away] | 13:28 | |
| anteaya | sdague: ah okay, sorry I got confused | 13:28 |
| AJaeger | rhochmuth: manually abort all of them... | 13:28 |
| jroll | +1 for let zuul do it's thing | 13:28 |
| sdague | AJaeger: but that doesn't really cancel them | 13:28 |
| jroll | and ptl or someone approve them all | 13:28 |
| *** yottatsa has quit IRC | 13:28 | |
| anteaya | rhochmuth: please don't do anything yet | 13:28 |
| *** tonytan4ever has joined #openstack-infra | 13:28 | |
| dhellmann | rhochmuth : at this point we're considering just letting you approve them all to push them in, but as anteaya says don't do anything until we agree on a plan | 13:28 |
| *** salv-orlando has joined #openstack-infra | 13:28 | |
| *** _amrith_ is now known as amrith | 13:29 | |
| rhochmuth | ok, | 13:29 |
| AJaeger | Let's just all call it an early weekend ;) | 13:29 |
| sdague | anteaya: so, on the let zuul do it's job front, rhochmuth approving the first 10, figure out if they merge in any reasonable time | 13:29 |
| sdague | if they do, approving them all through is fine | 13:29 |
| sdague | if they don't, because things are super gummed up, then something more drastic can be done | 13:29 |
| AJaeger | But would rhochmuth approve them? He said "I was expecting one commit"... | 13:29 |
| dhellmann | I like that. I might start with fewer than 10, though | 13:29 |
| sdague | AJaeger: I hope so, they are his commits from his repo | 13:30 |
| sdague | the whole point was to keep history | 13:30 |
| rhochmuth | so this is a new repo and I don't have +2 privs on it | 13:30 |
| anteaya | I'm not following the approve them approach as that just keeps us gummed up rather than letting the check queue subside, they still have to finish in check | 13:30 |
| *** ddieterly[away] is now known as ddieterly | 13:30 | |
| sdague | anteaya: most of them are done in check | 13:30 |
| sdague | they are all noop | 13:30 |
| *** salv-orlando has quit IRC | 13:31 | |
| *** thorst_ has joined #openstack-infra | 13:31 | |
| anteaya | great, then I'm for just leave them be for now and let this subside | 13:31 |
| AJaeger | https://review.openstack.org/#/admin/groups/1370,members has no members ;( | 13:31 |
| sdague | https://review.openstack.org/#/c/313318/1 jenkins +1 | 13:31 |
| anteaya | not send them back in with approvals | 13:31 |
| dhellmann | pabelanger : can you fix the gerrit permissions on this new repo? ^^ | 13:31 |
| sdague | anteaya: the reason zuul is slow in check is because it's processing the 177 dependencies on each of these | 13:31 |
| AJaeger | rhochmuth, davidSubirosPere : What every you do: please don't push the series again... | 13:32 |
| sdague | as the number of dependencies goes down | 13:32 |
| sdague | they get faster | 13:32 |
| pabelanger | dhellmann: sure, I just need a gerrit id | 13:32 |
| anteaya | sdague: wonderful | 13:32 |
| sdague | hence the "merge some stuff that jenkins already +1ed" | 13:32 |
| davidSubirosPere | AJaeger, I won't, sorry again | 13:32 |
| anteaya | yes zuul stats in cacti confirm zuul is recovering | 13:32 |
| davidSubirosPere | for now I don't do anything, let me know if I have to abort | 13:32 |
| davidSubirosPere | sorry, I mean abandon | 13:32 |
| sdague | davidSubirosPere: definitely don't abandon at this point | 13:33 |
| AJaeger | rhochmuth: what's your email - so that pabelanger can add you as initial member to the repo? | 13:33 |
| davidSubirosPere | ok | 13:33 |
| sdague | it looks like it's +1ed at least the first 100 changes | 13:33 |
| *** nadya has joined #openstack-infra | 13:33 | |
| rhochmuth | AJaeger: roland.hochmuth@hp.com | 13:33 |
| sdague | so doing the first 10 will give an idea of how that would proceed | 13:33 |
| pabelanger | AJaeger: rhochmuth added | 13:34 |
| AJaeger | thanks, pabelanger . rhochmuth you're setup now... | 13:34 |
| *** hdaniel has joined #openstack-infra | 13:34 | |
| rhochmuth | can you also add me on monasca-transform too? | 13:34 |
| *** BigWillie has joined #openstack-infra | 13:34 | |
| AJaeger | rhochmuth: URL to members? | 13:34 |
| AJaeger | sdague: indeed, first 100+ jobs are done, so let's wait... | 13:35 |
| *** javeriak_ has joined #openstack-infra | 13:35 | |
| anteaya | I'm for waiting | 13:35 |
| sdague | AJaeger: I don't understand the waiting | 13:35 |
| sdague | what are you waiting for? | 13:35 |
| rhochmuth | AJaeger: not sure what you mean? | 13:35 |
| anteaya | according to cacti zuul can't process anything right now | 13:35 |
| *** thorst_ has quit IRC | 13:35 | |
| dhellmann | rhochmuth : please let's fix this one before dealing with the other repo. can you approve those first 10 patches/ | 13:35 |
| dhellmann | ? | 13:35 |
| anteaya | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=397&rra_id=all | 13:35 |
| rhochmuth | sure | 13:36 |
| pabelanger | What I am curious about, if patch B is in check (it has already been handled my zuul-merger), if patch A then is +A and merged, I don't think patch B will be affected if already in the check queue. And still carry the initial dependency chain. | 13:36 |
| AJaeger | sdague: Not abort or anything - let zuul do it's job... | 13:36 |
| *** yamamoto has quit IRC | 13:36 | |
| sdague | AJaeger: right, I agree not aborting | 13:36 |
| sdague | my suggestion was to start approving those patches | 13:37 |
| AJaeger | rhochmuth: > | 13:37 |
| AJaeger | rhochmuth: > | 13:37 |
| *** thorst_ has joined #openstack-infra | 13:37 | |
| AJaeger | rhochmuth: sorry ^ let me try again to paste... | 13:37 |
| pabelanger | but like anteaya said, I'm leaning to get zuul do its job and process the patches. Since we are week 1 after summit. | 13:37 |
| sdague | pabelanger: right, it will, unless there are hicups and it restarts the check for whatever reason | 13:37 |
| AJaeger | rhochmuth: https://review.openstack.org/#/admin/groups/1370,members is the URL to the monasca-analytics core group. What's the URL for the other group? | 13:37 |
| sdague | anyway, I need to drop off. Good luck folks. | 13:38 |
| *** hdaniel has quit IRC | 13:39 | |
| *** yaume has quit IRC | 13:39 | |
| anteaya | I think approving will just increase the load on zuul, not decrease it | 13:39 |
| *** yamamoto has joined #openstack-infra | 13:40 | |
| anteaya | I'm -1 on approving patches at the moment | 13:40 |
| *** Hal has joined #openstack-infra | 13:40 | |
| *** Hal is now known as Guest78215 | 13:40 | |
| *** thorst_ has quit IRC | 13:41 | |
| wznoinsk | hi infra, is there a zuul layout.yml check I could do except for yaml(lint) ? it looks like zuul doesn't start pipelinemanager at all when i have a mistake in the layout.yml (but I don't know what the mistake is after a lot of changes) | 13:41 |
| rhochmuth | AJaeger: https://review.openstack.org/#/admin/groups/1350,members | 13:42 |
| AJaeger | pabelanger: ^ | 13:43 |
| *** ifarkas has joined #openstack-infra | 13:43 | |
| AJaeger | wznoinsk: tox -e zuul in project-config parses the complete file... | 13:43 |
| *** sdague has quit IRC | 13:43 | |
| *** yamamoto has quit IRC | 13:44 | |
| wznoinsk | AJaeger: thanks a life saver | 13:44 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add a table for story permissions https://review.openstack.org/310951 | 13:44 |
| *** signed8bit is now known as signed8bit_Zzz | 13:45 | |
| pabelanger | It does seem there is a downside to letting zuul process openstack/monasca-analytics, the next patch in the check queue is 230434,22 and has been idle for 3hrs4mins. Meaning, nodepool has not launched any jobs for that patch. | 13:45 |
| pabelanger | So, any new patch uploaded into zuul is blocked it seems | 13:46 |
| *** signed8bit_Zzz is now known as signed8bit | 13:46 | |
| *** javeriak_ has quit IRC | 13:46 | |
| AJaeger | pabelanger: I'm fine with a #status notice "Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend!" ;) | 13:47 |
| * AJaeger wouldn't object to other solutions either - I just don't know enough to advise here... | 13:47 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:48 | |
| *** ddieterly is now known as ddieterly[away] | 13:48 | |
| anteaya | I'm in favour of a status notice | 13:49 |
| mescanef | jhesketh: hello. regarding issue https://storyboard.openstack.org/#!/story/2000532 - in my env adding keepalive to paramikos transport in gerritwatcher solves the issue. related change is on review prepared by someone already but not yet merged: https://review.openstack.org/#/c/238988/ | 13:49 |
| anteaya | and we should have a few roots with some more experience here show up in the next hour so I'm in still in favour of waiting | 13:49 |
| dhellmann | are we still going to approve some of the patches to see how that goes? | 13:49 |
| pabelanger | AJaeger: anteaya: sure, let me see if status will pick it up | 13:49 |
| *** jaypipes has quit IRC | 13:50 | |
| pabelanger | #status notice Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend! | 13:50 |
| anteaya | dhellmann: I'm not in favour of that approach as I think that will just put more load on zuul, not decrease any load | 13:50 |
| dhellmann | anteaya : how so? | 13:50 |
| anteaya | you are just asking zuul to process them again | 13:50 |
| *** trown is now known as trown|outtypewww | 13:50 | |
| pabelanger | anteaya: dhellmann: see my comment above about new patches being blocked | 13:50 |
| *** thorst_ has joined #openstack-infra | 13:50 | |
| *** yamamoto has quit IRC | 13:51 | |
| anteaya | it processed 100 patches, according to someone who posted in backscroll, zuul now doesn't have to process those patches, if you approve them you are asking zuul to care about them again | 13:51 |
| pabelanger | AJaeger: guess I don't have permission | 13:51 |
| dhellmann | anteaya : ok. so we wait for them to pass check, then abandon them? | 13:51 |
| dhellmann | if this is preventing folks from getting any work done I think we need to take stronger action. if we have to wait a bit for more experience to come online, that's fine | 13:51 |
| anteaya | pabelanger: right that is what I'm saying, putting patches into the gate queue are blocked | 13:51 |
| anteaya | zuul isn't processing new patches at this time | 13:52 |
| anteaya | dhellmann: I don'tknow what the step is after they clear check | 13:52 |
| anteaya | after they clear check I see no advantage in abandoning them | 13:52 |
| anteaya | zuul doesn't care about them after they have left the check queue | 13:52 |
| anteaya | so abandoning them serves no purpose as far as un gumming zuul | 13:53 |
| dhellmann | I suppose at that point if they are approved one at a time it wouldn't cause a problem | 13:53 |
| anteaya | I haven't even thought about how to fix the repo | 13:53 |
| anteaya | possibly? | 13:53 |
| anteaya | I'm trying to figure out the best path for clearing zuul | 13:53 |
| anteaya | and zuul has been great at chugging along through things, it doesn't tend to fall over | 13:54 |
| pabelanger | Right, but knowing that now, I'm thinking maybe we do need stronger action like dhellmann is suggesting. Since zuul is just CPU pegged doing noop checks, I'm not sure letting the check queue back up is in our best interest | 13:54 |
| anteaya | and according to cacti it has lots of free memory | 13:54 |
| anteaya | pabelanger: okay, I'm still of a mind fungi or clarkb will show up soon | 13:54 |
| dhellmann | if we have a 3+hr wait for a new job to even start, we're going to pretty quickly get to a point where nothing happens today | 13:55 |
| anteaya | but if you would like to move ahead, which direction are you leaning? | 13:55 |
| anteaya | dhellmann: I also recognize that currently pabelanger is the only root right now | 13:55 |
| *** edmondsw has joined #openstack-infra | 13:55 | |
| *** yamamoto has joined #openstack-infra | 13:55 | |
| dhellmann | sure. I'd like to have a concrete proposal to put to fungi or clarkb when they come online | 13:55 |
| pabelanger | anteaya: I'd like to get input from fungi / clarkb too. | 13:55 |
| anteaya | and he is doing a great job but still would benefit from consultations with peers if they show up soon | 13:55 |
| anteaya | pabelanger: me too | 13:56 |
| anteaya | okay so what's on the table? | 13:56 |
| anteaya | what do folks want to do? | 13:56 |
| *** ddieterly[away] is now known as ddieterly | 13:56 | |
| pabelanger | I think we have 2 options, either dump the check / gate queue, restart zuul. then requeue only gate. Or +A a patch to see how zuul response | 13:56 |
| anteaya | okay | 13:56 |
| dhellmann | I think I agree with anteaya that approving patches doesn't buy us anything right now | 13:56 |
| AJaeger | pabelanger: are you authenticated? | 13:56 |
| *** dansmith is now known as snotdan | 13:56 | |
| anteaya | of those two options, I'm in the restart zuul camp, save the gate queue | 13:57 |
| pabelanger | There could be more options, but I don't know them at this point in time | 13:57 |
| pabelanger | AJaeger: thought I was | 13:57 |
| anteaya | pabelanger: have you restarted zuul before? | 13:57 |
| AJaeger | pabelanger: yes, no permissions - see system-config/hiera/common.yaml | 13:57 |
| *** chuck__ is now known as zul | 13:57 | |
| *** akshai has quit IRC | 13:57 | |
| dhellmann | we did discuss a few options for causing these items to drop out of check without restarting zuul, but I think we ruled all of those out as not effective? | 13:57 |
| *** xarses has quit IRC | 13:58 | |
| anteaya | dhellmann: like abandoning patches? | 13:58 |
| pabelanger | anteaya: I'd be following http://docs.openstack.org/infra/system-config/zuul.html#restarts | 13:58 |
| dhellmann | abandoning, or changing their commit message | 13:58 |
| anteaya | dhellmann: I don't believe abandoning patches will affect check at all | 13:58 |
| dhellmann | ok | 13:58 |
| *** jkraj has joined #openstack-infra | 13:58 | |
| anteaya | pabelanger: right so that's no, you haven't restarted it before | 13:58 |
| anteaya | pabelanger: which is fine | 13:58 |
| *** snotdan is now known as superdan | 13:58 | |
| anteaya | and I'm fine with restart as a way forward | 13:59 |
| pabelanger | anteaya: sorry, I have. With fungi, pleia2. but I would be following the steps in the document | 13:59 |
| anteaya | but I do think that waiting another 40 minutes for some backup for pabelanger is in our best interest | 13:59 |
| *** akshai has joined #openstack-infra | 13:59 | |
| anteaya | pabelanger: ah okay thanks | 13:59 |
| *** jtomasek has joined #openstack-infra | 13:59 | |
| anteaya | pabelanger: your call then | 13:59 |
| anteaya | I'm leaning towards waiting | 13:59 |
| anteaya | other than blocking normal friday work, there is no critical release | 14:00 |
| *** vhosakot has joined #openstack-infra | 14:00 | |
| anteaya | this is not a make or break moment | 14:00 |
| anteaya | just so you have some backup on the restart pabelanger | 14:00 |
| anteaya | but up to you | 14:00 |
| *** sdake_ has joined #openstack-infra | 14:00 | |
| *** Goneri has quit IRC | 14:01 | |
| pabelanger | we're this far into the delay, I have no issue waiting for fungi or clarkb | 14:01 |
| anteaya | pabelanger: I support that decision | 14:01 |
| pabelanger | going to see if I have an out of band contain for either from the summit | 14:01 |
| dhellmann | wfm | 14:01 |
| anteaya | dhellmann: thanks :) glad you have your help here | 14:01 |
| anteaya | pabelanger: okay | 14:02 |
| *** yamamoto has quit IRC | 14:02 | |
| *** yamamoto has joined #openstack-infra | 14:02 | |
| *** maestro has quit IRC | 14:03 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu binary gate on liberty branch https://review.openstack.org/313561 | 14:04 |
| *** sdake has quit IRC | 14:04 | |
| *** javeriak has joined #openstack-infra | 14:04 | |
| *** psilvad has quit IRC | 14:06 | |
| senk | zuul seems really fucked up :D | 14:06 |
| *** ayoung has joined #openstack-infra | 14:07 | |
| senk | should also start plan how to avoid that in the future.. | 14:07 |
| *** ramishra has quit IRC | 14:07 | |
| *** ramishra has joined #openstack-infra | 14:07 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run cinderclient functional tests in Identity v3-only https://review.openstack.org/313562 | 14:08 |
| anteaya | senk: can you find better ways to express yourself please? | 14:08 |
| senk | anteaya: i'm currently thinking about how that desaster could have been avoided. | 14:09 |
| anteaya | I'm currently thinking that there is enough stress in channel without you swearing | 14:09 |
| *** mtanino has joined #openstack-infra | 14:09 | |
| anteaya | I find your choice of language offensive | 14:09 |
| *** jang has joined #openstack-infra | 14:09 | |
| *** sridhar_ram has joined #openstack-infra | 14:10 | |
| senk | anteaya: sorry about that :-/ it was not my intention to offend anyone | 14:10 |
| pabelanger | AJaeger: thanks, adding it now | 14:10 |
| anteaya | well work on that goal | 14:10 |
| openstackgerrit | Zane Bitter proposed openstack/gertty: Support SyncQueriedChanges batching in Gerrit >=2.9 https://review.openstack.org/313563 | 14:10 |
| *** jkraj has quit IRC | 14:11 | |
| *** nadya has quit IRC | 14:12 | |
| *** sridhar_ram1 has joined #openstack-infra | 14:14 | |
| *** pabelanger has quit IRC | 14:14 | |
| *** pabelanger has joined #openstack-infra | 14:14 | |
| pabelanger | #status notice Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend! | 14:14 |
| openstackstatus | pabelanger: sending notice | 14:14 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow the creation of private stories https://review.openstack.org/310952 | 14:14 |
| *** Goneri has joined #openstack-infra | 14:14 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add a table for story permissions https://review.openstack.org/310951 | 14:14 |
| *** sridhar_ram has quit IRC | 14:14 | |
| pabelanger | AJaeger: I needed to authenticate with freenode | 14:15 |
| AJaeger | Ah! | 14:15 |
| * mordred would like to tip his hat to zuul for dealing with this number of changes, even if slowly | 14:15 | |
| -openstackstatus- NOTICE: Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend! | 14:15 | |
| anteaya | zuul io seems to have come back down to pre-spike levels: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=410&rra_id=all | 14:15 |
| *** salv-orlando has joined #openstack-infra | 14:15 | |
| anteaya | mordred: morning | 14:15 |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Retire openstack/designate-msdnsagent repo https://review.openstack.org/313565 | 14:15 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add Emilien Macchi blog to TripleO blogs feed https://review.openstack.org/313566 | 14:15 |
| AJaeger | and you were listed in system-config, not sure how I could miss that ;( | 14:15 |
| anteaya | mordred: isn't zuul a trooper! | 14:16 |
| mordred | anteaya: right? | 14:16 |
| * regXboi watches and reads, waiting for patches behind the elephant to finally make it into the python | 14:16 | |
| anteaya | it is a sight to behold | 14:16 |
| anteaya | regXboi: hey there, some good reading | 14:16 |
| regXboi | anteaya: ack - I wasn't going to say anything unless things started to go off the rails :) | 14:16 |
| *** yaume has joined #openstack-infra | 14:17 | |
| openstackstatus | pabelanger: finished sending notice | 14:17 |
| anteaya | mordred: so the plan is pabelanger restarts zuul and saves the gate queue and not the check queue | 14:17 |
| * regXboi sighs | 14:17 | |
| anteaya | mordred: are you available to be his backup as he does this? | 14:17 |
| *** links has quit IRC | 14:17 | |
| anteaya | regXboi: ha ha ha, ah thank you | 14:17 |
| * regXboi prepares rechecks on patches waiting in queue | 14:17 | |
| *** vgridnev has joined #openstack-infra | 14:17 | |
| anteaya | regXboi: let zuul take a breath before you send them all back in? | 14:18 |
| *** xarses has joined #openstack-infra | 14:18 | |
| anteaya | regXboi: just for a moment? | 14:18 |
| regXboi | anteaya: it's only two - I haven't pushed any more and I was going to wait until after my phone call... which could take who knows how long :) | 14:18 |
| anteaya | ah the phone call | 14:18 |
| anteaya | fair enough, thanks | 14:18 |
| regXboi | the *dreaded* *unscheduled* *massively delayed* phone call | 14:18 |
| *** jlibosva has joined #openstack-infra | 14:18 | |
| anteaya | wooo the best kind of phone call | 14:19 |
| regXboi | but who's counting :) | 14:19 |
| anteaya | :) | 14:19 |
| anteaya | mordred: what say you? | 14:19 |
| anteaya | you can hand pabelanger cats helpfully | 14:19 |
| *** javeriak has quit IRC | 14:19 | |
| *** vhosakot has quit IRC | 14:19 | |
| *** jaypipes has joined #openstack-infra | 14:20 | |
| mordred | anteaya: I am happy to hand cats | 14:21 |
| anteaya | pabelanger: are you willing to proceed with the zuul restart with mordred as backup cat distributor? | 14:22 |
| pabelanger | sure, we all agree, dump check / gate queues, restart zuul, restore only gate | 14:22 |
| anteaya | agreed | 14:22 |
| * mordred hands pabelanger a few cats that are only slightly ravenous | 14:22 | |
| anteaya | so yes save the gate queue before zuul is stopped | 14:23 |
| *** zz_dimtruck is now known as dimtruck | 14:23 | |
| * regXboi mutters ... speaking of ravenous ... time to grab something for breakfast | 14:23 | |
| anteaya | regXboi: happy breakfast | 14:23 |
| pabelanger | okay, fetching documentation | 14:23 |
| anteaya | pabelanger: very good | 14:23 |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Remove Ceilometer from projects https://review.openstack.org/311989 | 14:24 |
| AJaeger | pabelanger: the top two jobs in the gate are nearly finished, let's merge those first... | 14:24 |
| pabelanger | okay, zuul stopped | 14:25 |
| pabelanger | deleted unused nodes in nodepool | 14:25 |
| AJaeger | then let's restart them - no problem... | 14:25 |
| anteaya | AJaeger: yeah only so many things he can juggle at a time | 14:25 |
| *** ccamacho has quit IRC | 14:25 | |
| *** r-daneel has joined #openstack-infra | 14:26 | |
| pabelanger | zuul restarted | 14:26 |
| AJaeger | anteaya: bad timing;) No problem... | 14:26 |
| anteaya | AJaeger: :) good intentions though | 14:26 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Increase timeout and logging for crm_resource https://review.openstack.org/313011 | 14:27 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use megabytes for overcloud image size https://review.openstack.org/312594 | 14:27 |
| anteaya | looks good from status.o.o/zuul | 14:28 |
| pabelanger | okay, gate requeued | 14:29 |
| pabelanger | checking nodepool | 14:29 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only https://review.openstack.org/313572 | 14:29 |
| *** ccamacho has joined #openstack-infra | 14:29 | |
| *** javeriak has joined #openstack-infra | 14:30 | |
| pabelanger | okay, nodepool building nodes | 14:31 |
| pabelanger | I think we are good | 14:31 |
| anteaya | well done | 14:31 |
| anteaya | mordred: thanks for being available | 14:31 |
| AJaeger | thanks, pabelanger ! | 14:31 |
| pabelanger | just confirming we dont | 14:31 |
| pabelanger | have any NOT_REGISTERED jobs | 14:31 |
| anteaya | good call | 14:31 |
| AJaeger | pabelanger: please send out a #status notice and ask for recheck of check jobs | 14:32 |
| senk | great work pabelanger ! | 14:32 |
| fungi | guess i picked a fine time to take a late morning. catching up on the situation now | 14:32 |
| anteaya | fungi: pabelanger has it under control | 14:32 |
| mordred | pabelanger: nicely done | 14:33 |
| anteaya | fungi: the tl;dr is someone pushed a 177 patch chain which slowed down zuul | 14:33 |
| * mordred hands pabelanger a fluffy cat that has been recently fed | 14:33 | |
| openstackgerrit | Jianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool https://review.openstack.org/310698 | 14:33 |
| anteaya | fungi: after some discussion we agreed the best way forward was to restart zuul saving only the gate queue | 14:33 |
| pabelanger | I don't see any NOT_REGISTERED, so think we are in the clear | 14:33 |
| anteaya | pabelanger: thanks for checking | 14:33 |
| pabelanger | anteaya: AJaeger: mordred: senk: thanks | 14:33 |
| *** psilvad has joined #openstack-infra | 14:34 | |
| jroll | pabelanger: ^5, thanks for the help | 14:34 |
| rhochmuth | thanks everyone | 14:34 |
| mordred | anteaya, pabelanger: in the future there is a way that infra-core people can help with importing 177 patch chains without triggering tons of zuul, in case that's helpful to know | 14:34 |
| *** kushal has quit IRC | 14:35 | |
| mordred | rhochmuth: if it makes you feel any better, the first person to do what you did was vish | 14:35 |
| pabelanger | so, I think moving forward, rhochmuth will need to +A the current +1 patches | 14:35 |
| anteaya | rhochmuth: okay so, were you part of the decision to submit a 177 patch chain? | 14:35 |
| fungi | also, glad to see this was not an emergency, just zuul working as expected | 14:35 |
| pabelanger | mordred: agreed! | 14:35 |
| mordred | rhochmuth: which is to say - it happens to the best of us | 14:35 |
| rhochmuth | mordred: that helps, thanks | 14:35 |
| anteaya | rhochmuth: if yes, did anyone involved read any documentation prior to that decision? | 14:35 |
| fungi | an excellent test of its ability to handle very, very long patch series | 14:35 |
| mordred | fungi: right? | 14:35 |
| *** rbrndt has joined #openstack-infra | 14:35 | |
| anteaya | fungi: yes exactly | 14:35 |
| mordred | I'm not sure I think we need to +A those patches | 14:35 |
| mordred | rhochmuth: this is import of history, yeah? | 14:36 |
| rhochmuth | i'm not sure it is | 14:36 |
| rhochmuth | i'll have to work with davidSubirosPere | 14:36 |
| anteaya | rhochmuth: I'm trying to understand if changing documentation would do any good, which wouldn't if you didn't look at any | 14:36 |
| *** thorst_ has quit IRC | 14:36 | |
| rhochmuth | i was only expecting on e commit | 14:36 |
| *** tojuvone_ has quit IRC | 14:36 | |
| rhochmuth | not the entire history too | 14:36 |
| anteaya | rhochmuth: you just thought this would end up as one patch? | 14:36 |
| mordred | rhochmuth: gotcha. so - if the patches in question are correct | 14:36 |
| AJaeger | rhochmuth: next time: If you create a repo, you can import an existing one. So, if there's one you want to import with history, it's trivial at repo creation time... | 14:37 |
| *** dprince has quit IRC | 14:37 | |
| mordred | rhochmuth: and we do want to land them - I suggest that we have an infra-core person just push them to the branch, which will close all of the reviews without making zuul run any gate jobs for them | 14:37 |
| * fungi is surprised git review didn't throw a warning like "you are about to submit 176 changes..." | 14:37 | |
| rhochmuth | the repo wasn't in public github to import | 14:37 |
| pabelanger | fungi: I was thinking that too | 14:37 |
| mordred | rhochmuth: which is easy to do | 14:37 |
| pabelanger | fungi: maybe we need clippy to pop up asking are you sure | 14:38 |
| mordred | so, basically, let's just connect once you have an idea of what you want the end-state to be and we can help you get there | 14:38 |
| rhochmuth | ok | 14:38 |
| fungi | doesn't have to be github either. i just temporarily chuck the git repo up onto my personal webserver until it gets imported, then delete it | 14:38 |
| anteaya | pabelanger: no, no clippy, no clippy | 14:38 |
| rhochmuth | davidsupirospere is new to the project from hp labs | 14:38 |
| jroll | mordred: fwiw, there's no jobs here, just the explosion of events | 14:38 |
| rhochmuth | but will discuss options with him | 14:38 |
| AJaeger | rhochmuth: happy mentoring! | 14:38 |
| mordred | jroll: it's all no-op jobs? | 14:38 |
| jroll | mordred: yep | 14:38 |
| rhochmuth | i'm not a good mentor | 14:38 |
| rhochmuth | davidsubirospere are you still there | 14:39 |
| AJaeger | rhochmuth: then best find one ;) | 14:39 |
| mordred | jroll: k. well, still - one git push is still way less work than someone pressing +A 176 times | 14:39 |
| mordred | even with gertty | 14:39 |
| jroll | mordred: totes, just a heads up :) | 14:39 |
| pabelanger | I guess we should send a status notice telling people zuul was restarted and check queue lost | 14:39 |
| pabelanger | and to recheck patches | 14:39 |
| jroll | mordred: shocking that there's no mass approve scripts out there | 14:39 |
| fungi | jroll: actually... gertty has batch operations | 14:40 |
| anteaya | pabelanger: agreed | 14:40 |
| *** thorst_ has joined #openstack-infra | 14:40 | |
| fungi | flag the dependent series of reviews, batch approve/verify+2/review+2/submit | 14:40 |
| *** MarkAtwood has joined #openstack-infra | 14:40 | |
| * fungi can't remember if batch submit is implemented yet though | 14:41 | |
| *** hdd has joined #openstack-infra | 14:41 | |
| anteaya | I thought it was | 14:41 |
| * dhellmann makes a note to give gertty another try | 14:41 | |
| anteaya | I thought jeblair batch approved something a month or so ago | 14:41 |
| dhellmann | I assume after the restart it's safe to do things like tag releases? | 14:42 |
| anteaya | dhellmann: yes, you may operate as normal now | 14:42 |
| dhellmann | anteaya : thanks for confirming | 14:42 |
| fungi | review, abandon/restore, edit topic | 14:42 |
| fungi | no batch submit (yet) | 14:42 |
| anteaya | dhellmann: thanks for your help here | 14:42 |
| davidSubirosPere | rhochmuth yes I'm here | 14:43 |
| anteaya | ah, sorry it was batch review I was referencing | 14:43 |
| *** e0ne has quit IRC | 14:43 | |
| fungi | not that we use submit very often since it bypasses ci completely, so maybe not critical to have a batch version (also maybe even a little dangerous) | 14:43 |
| anteaya | davidSubirosPere: out of curiosity, did you reference any documentation prior to submitting that patch chain? | 14:43 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add magnum services call to shade https://review.openstack.org/313583 | 14:43 |
| *** timsim has joined #openstack-infra | 14:43 | |
| *** e0ne has joined #openstack-infra | 14:44 | |
| anteaya | davidSubirosPere: I ask wondering if changing our documentation would have made any difference here | 14:44 |
| *** Na3iL has quit IRC | 14:44 | |
| rhochmuth | davidSubirosPere: everything is restored, we should talk offline about doing a single review | 14:44 |
| fungi | and anyway, batch submit could just be accomplished with a push to the branch instead of refs/for/refs as mordred suggested | 14:44 |
| fungi | by one of the gerrit admins | 14:44 |
| davidSubirosPere | I checked the documentation to create the account and submit a git review, but maybe I should have checked more documentation before pushing all those commits | 14:44 |
| pabelanger | #status notice Zuul has been restarted. As a results, we only preserved patches in the gate queue. Be sure to recheck your patches in gerrit if needed. | 14:45 |
| openstackstatus | pabelanger: sending notice | 14:45 |
| rhochmuth | fungi: the repo is private, so not sure can do a batch push | 14:45 |
| fungi | rhochmuth: well, it's not intended to be private, would just need to get a copy of it | 14:45 |
| *** yottatsa has joined #openstack-infra | 14:46 | |
| -openstackstatus- NOTICE: Zuul has been restarted. As a results, we only preserved patches in the gate queue. Be sure to recheck your patches in gerrit if needed. | 14:46 | |
| fungi | davidSubirosPere: we might could also make the multiple changes warning/confirmation git-review displays a little more verbose/scary | 14:46 |
| sc68cal | lol love that status update | 14:46 |
| anteaya | sc68cal: you have pabelanger to thank for that | 14:46 |
| davidSubirosPere | yes, I think it would be a good idea | 14:46 |
| anteaya | sc68cal: as well as fixing zuul | 14:46 |
| *** mixos has joined #openstack-infra | 14:46 | |
| davidSubirosPere | so people know that a job will be created for each commit | 14:47 |
| sc68cal | sorry, meant the previous one that said "Have a great weekend" | 14:47 |
| *** vgridnev has quit IRC | 14:47 | |
| anteaya | sc68cal: ah, I think that was AJaeger's composition | 14:47 |
| openstackstatus | pabelanger: finished sending notice | 14:47 |
| sc68cal | I mean it is true, it's going to take a loong time for zuul to right itself, so might as well have an early friday ;) | 14:47 |
| *** yamamoto has quit IRC | 14:47 | |
| fungi | also, in the future, it's possible to filter the exported pipeline contents to filter out unwanted changes. for example, could just | grep -v problem_projectname | 14:47 |
| *** psilvad has quit IRC | 14:48 | |
| anteaya | davidSubirosPere: have you taken a look at our developer's guide yet? it might interest you: http://docs.openstack.org/infra/manual/developers.html | 14:48 |
| mordred | fungi: we can get a copy of it with git review -d | 14:48 |
| anteaya | sc68cal: well pabelanger restarted it | 14:48 |
| pabelanger | fungi: Agreed. Was thinking of updating our documentation about that | 14:48 |
| fungi | mordred: good point, now that it's in gerrit | 14:48 |
| davidSubirosPere | thanks! I had a quick look, I will have another look | 14:48 |
| mordred | fungi: so all a gerrit admin needs to do is git review -d the end of the series, then git push gerrit master | 14:48 |
| anteaya | sc68cal: so pick your status update of choice | 14:48 |
| sc68cal | I'll take the one where I get to start my weekend early :) | 14:48 |
| mordred | sc68cal: well chosen | 14:49 |
| fungi | sc68cal: i thought i had started my weekend early, then i made the mistake of looking in irc ;) | 14:49 |
| * mordred on the other hand is going to go to the driver's license office | 14:49 | |
| *** nadya has joined #openstack-infra | 14:49 | |
| mordred | fungi: NO VACATION FOR YOU | 14:49 |
| anteaya | davidSubirosPere: that is the documentation that should discuss our testing system and that each patch is tested individually | 14:49 |
| sc68cal | fungi: know that feeling. :) | 14:49 |
| fungi | mordred: always makes me think of "dmv" by primus | 14:49 |
| *** ewilson has quit IRC | 14:50 | |
| mordred | anteaya: I think it might be useful to add something to the import-a-project docs | 14:50 |
| davidSubirosPere | http://docs.openstack.org/infra/manual/developers.html#squashing-changes I see it's explained there | 14:50 |
| anteaya | mordred: fair enough | 14:50 |
| mordred | anteaya: that covers the "what if the repo you want to import currently only exists in a private location" | 14:50 |
| anteaya | davidSubirosPere: ah lovely, glad you found it | 14:50 |
| mordred | since that is what I believe tripped up davidSubirosPere and rhochmuth | 14:50 |
| anteaya | mordred: I see, yes | 14:50 |
| mordred | and I think we can make it clear that that is supported and there is a way to deal with it | 14:51 |
| anteaya | to be honest this wasn't a usecase I personally had considered | 14:51 |
| mordred | yah | 14:51 |
| fungi | anyway, what was the project and which change was intended as the topic branch tip? | 14:51 |
| mordred | which is why I think our docs do a bad job of helping people on it :) | 14:51 |
| anteaya | but agreed, making it clear is helpful | 14:51 |
| anteaya | fungi: monasca-analytics | 14:51 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in keystone https://review.openstack.org/311169 | 14:51 |
| anteaya | I don't know the branch tip change | 14:51 |
| fungi | i should be able to find it easily enough | 14:51 |
| fungi | thanks | 14:51 |
| anteaya | thank you | 14:52 |
| mordred | fungi: now, if only gerrit had a good way of finding the tip | 14:52 |
| fungi | gertty does | 14:52 |
| mordred | https://review.openstack.org/#/c/313494/ | 14:52 |
| mordred | there it is | 14:52 |
| fungi | gerrit always confuses me as to which end is which | 14:52 |
| mordred | it actually wasn't as bad in gerrit as I thought | 14:52 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 14:52 |
| fungi | heh... Related Changes (177) | 14:52 |
| mordred | (it helps that commit #1 had the subject "initial commit") | 14:52 |
| mordred | the related changes field holds up | 14:53 |
| anteaya | initial commit is a helpful initial commit title | 14:53 |
| mordred | fungi: how is gertty's display of it? | 14:53 |
| mordred | fungi: it might be a fun pathological test case for jeblair for gertty ui | 14:53 |
| anteaya | ha ha ha | 14:53 |
| fungi | i have to wait for it to sync since i wasn't subscribed yet | 14:54 |
| anteaya | ah | 14:54 |
| fungi | it's pulling them down pretty quickly though | 14:54 |
| anteaya | yay | 14:54 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only https://review.openstack.org/313588 | 14:54 |
| dhellmann | does anyone have any idea why a patch to openstack/releases would be showing up in the check-tripleo queue in the zuul web ui? there are no jobs listed in the little box on the web page https://review.openstack.org/#/c/311123/ | 14:54 |
| anteaya | oh while we are here, /home/gerrit2 on review.o.o is looking a bit getting-full: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all | 14:55 |
| *** esikachev has quit IRC | 14:55 | |
| *** jaypipes has quit IRC | 14:55 | |
| mordred | dhellmann: that is a fascinating question | 14:56 |
| fungi | dhellmann: i don't see it... on the page | 14:56 |
| *** vgridnev has joined #openstack-infra | 14:56 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 14:56 |
| dhellmann | fungi : it disappeared as soon as I asked the question | 14:56 |
| anteaya | dhellmann: I see 311123 in the check queue of status.o.o/zuul, 4 jobs all queued | 14:56 |
| fungi | dhellmann: oh | 14:56 |
| dhellmann | heisenbug? | 14:56 |
| fungi | maybe a rendering error? | 14:56 |
| anteaya | today seems to be the day | 14:56 |
| dhellmann | maybe that's all it was -- there were no jobs, so it seemed extra odd. | 14:56 |
| dhellmann | if it shows up again I'll mention it again | 14:57 |
| anteaya | hmmmmm | 14:57 |
| anteaya | thanks | 14:57 |
| anteaya | what happens when /home/gerrit2 on review.o.o fills up? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all | 14:57 |
| *** eharney has joined #openstack-infra | 14:57 | |
| *** sdake_ has quit IRC | 14:57 | |
| fungi | anteaya: badnez | 14:57 |
| anteaya | ah | 14:57 |
| anteaya | shall we wait and see how bad | 14:57 |
| anteaya | or shall we work it into a todo list? | 14:58 |
| mordred | anteaya: I vote on waiting and seeing what breaks | 14:58 |
| *** andrewbogott has quit IRC | 14:58 | |
| *** andrewbogott has joined #openstack-infra | 14:58 | |
| anteaya | mordred: that is to be expected | 14:58 |
| anteaya | mordred: since I believe you will be discussing driver license things when that happens | 14:58 |
| fungi | i'm looking now to see what's eating it all | 14:59 |
| anteaya | fungi: you are so wonderful | 14:59 |
| anteaya | thank you | 14:59 |
| fungi | unsurprisingly, it's pretty much all in the git repos | 15:01 |
| anteaya | okay | 15:01 |
| *** akshai has quit IRC | 15:01 | |
| anteaya | so make more space? | 15:01 |
| *** jaosorior has quit IRC | 15:02 | |
| fungi | yeah, but i'm digging deeper first | 15:02 |
| *** smcginnis_ has joined #openstack-infra | 15:02 | |
| anteaya | fair enough | 15:02 |
| *** Guest42930 has quit IRC | 15:05 | |
| *** david-lyle has joined #openstack-infra | 15:06 | |
| *** ociuhandu has joined #openstack-infra | 15:07 | |
| *** banix has quit IRC | 15:08 | |
| anteaya | AJaeger: thanks for creating 310929 | 15:09 |
| anteaya | AJaeger: I just added a comment about confirming source code prior to repo creation | 15:09 |
| fungi | nova's git repo is now up to 30gb on disk. i'm comparing that against the copy jeblair made back in early january | 15:09 |
| *** vgridnev has quit IRC | 15:09 | |
| fungi | looking at the disk utilization graph, we started a steady climb right around the gerrit 2.11 upgrade | 15:10 |
| *** sridhar_ram1 has quit IRC | 15:10 | |
| anteaya | oh my | 15:10 |
| fungi | leading me to believe gerrit 2.11 is stashing lots of new something in there 2.8 didn't | 15:10 |
| fungi | which may also explain why they added a background git gc task | 15:11 |
| *** jaypipes has joined #openstack-infra | 15:11 | |
| amrith | hi, I saw the note earlier about zuul being backed up, are we past that now? | 15:11 |
| anteaya | the mystery deepens | 15:11 |
| fungi | amrith: yes | 15:11 |
| amrith | hi fungi ... I've tried to recheck a review a couple of times and I see nothing in the zuul queues (see 303671) | 15:11 |
| amrith | ah wait | 15:12 |
| *** vgridnev has joined #openstack-infra | 15:12 | |
| amrith | I guess it is taking a while ... | 15:12 |
| amrith | now it shows up, thanks | 15:12 |
| amrith | for a while I didn't even see an openstack/trove queue | 15:13 |
| *** vgridnev has quit IRC | 15:13 | |
| *** r-mibu has quit IRC | 15:13 | |
| *** jamesmca_ has joined #openstack-infra | 15:13 | |
| *** r-mibu has joined #openstack-infra | 15:13 | |
| *** yamamoto has joined #openstack-infra | 15:14 | |
| *** mixos has quit IRC | 15:14 | |
| *** jlibosva has quit IRC | 15:15 | |
| *** sdake has joined #openstack-infra | 15:16 | |
| *** vgridnev has joined #openstack-infra | 15:16 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 15:17 |
| *** esikachev has joined #openstack-infra | 15:17 | |
| *** dizquierdo has quit IRC | 15:17 | |
| AJaeger | sc68cal: especially if you're in Europe ;) | 15:18 |
| *** arxcruz has quit IRC | 15:19 | |
| *** zhurong has quit IRC | 15:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add project-config reviewing guidelines https://review.openstack.org/310929 | 15:21 |
| AJaeger | anteaya: ready to merge now? ^ | 15:21 |
| *** dynarro has joined #openstack-infra | 15:21 | |
| AJaeger | fungi, I'd like to get my bonus points for not storing pot files in git anymore: Do you or any other project-config core time to review https://review.openstack.org/313114, please? | 15:22 |
| anteaya | AJaeger: I'm happy, thank you | 15:22 |
| AJaeger | Any other project-config cores that like to review our reviewing guidelines, please? https://review.openstack.org/310929 | 15:22 |
| AJaeger | thanks, anteaya ! | 15:22 |
| anteaya | thanks for writing it | 15:23 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow the creation of private stories https://review.openstack.org/310952 | 15:23 |
| fungi | okay, so the size of the nova repo on review.o.o was 5gb on january 8, and is 30gb now five months later | 15:24 |
| AJaeger | wow! | 15:24 |
| AJaeger | how much of that is due to pot source files? | 15:24 |
| anteaya | that seems not related to nova contributors | 15:24 |
| AJaeger | we could also switch to weekly imports ;) | 15:24 |
| anteaya | AJaeger: did you want to wait for fungi on 313114? | 15:24 |
| fungi | agreed. i think this is some sort of cruft building up in there | 15:24 |
| *** akshai has joined #openstack-infra | 15:25 | |
| *** vhosakot has joined #openstack-infra | 15:25 | |
| *** yamamoto has quit IRC | 15:25 | |
| fungi | probably not potfile updates | 15:25 |
| *** vgridnev has quit IRC | 15:25 | |
| AJaeger | anteaya: he mentioned in the prerequisite for this, the bonus points ;) So, I'm fine with merging it, just wanted it reviewed ;) fungi, do you want to review? | 15:25 |
| fungi | AJaeger: if it has enough reviews already, then not necessary | 15:26 |
| anteaya | I can merge it | 15:26 |
| AJaeger | fungi, it has now - thanks to anteaya ! | 15:26 |
| anteaya | +A | 15:26 |
| fungi | working on adding a second volume to review.o.o for now to buy us time to investigate the out of control growth in its git repos | 15:26 |
| anteaya | sounds good | 15:26 |
| AJaeger | the next translation updates will be large - they will remove pot files ;) But afterwards they will be significantly smaller... | 15:27 |
| pabelanger | clarkb: I am confused why nodepool didn't build new images at 10am UTC today | 15:28 |
| fungi | pabelanger: no logs? | 15:28 |
| *** jamesmca_ has quit IRC | 15:28 | |
| fungi | pabelanger: or started at 14:14 instead? | 15:29 |
| pabelanger | fungi: from what I see, they started at last night | 15:29 |
| pabelanger | fungi: we updated it to start at 10:34 now | 15:29 |
| pabelanger | but don't see builds at that time | 15:30 |
| fungi | and restarted the builder daemon after that landed? | 15:30 |
| pabelanger | yup | 15:30 |
| pabelanger | currently uploading an image to rax-iad, since it failed before | 15:31 |
| pabelanger | but, will restart again to be sure | 15:31 |
| fungi | yikes, our gerrit trove db is also almost out of space | 15:31 |
| *** jordanP has quit IRC | 15:31 | |
| anteaya | fungi: oh no | 15:31 |
| anteaya | fungi: glad you checked, is that on cacti anywhere? | 15:31 |
| fungi | nope | 15:32 |
| anteaya | ah | 15:32 |
| fungi | it was 10gb and 90% full, so i'm upping it to 50gb | 15:32 |
| anteaya | wow | 15:32 |
| *** sdake has quit IRC | 15:32 | |
| fungi | looks like it's supposed to be able to do that online | 15:32 |
| fungi | so hopefully non-impacting | 15:32 |
| *** dmk0202 has quit IRC | 15:33 | |
| *** jtomasek has quit IRC | 15:33 | |
| *** esikachev has quit IRC | 15:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Do not store source translation files in git https://review.openstack.org/313114 | 15:34 |
| pabelanger | mordred: when you have a moment, can you look into /var/log/nodepool/nodepool-builder.log on nodepool.o.o I am seeing a lot of Manager rax-ord running task ImageTaskGet spamming the lgs | 15:34 |
| pabelanger | logs* | 15:34 |
| *** dmk0202 has joined #openstack-infra | 15:34 | |
| *** mixos has joined #openstack-infra | 15:34 | |
| pabelanger | curious if that is to be expected | 15:35 |
| *** bpokorny has joined #openstack-infra | 15:35 | |
| *** dtantsur is now known as dtantsur|afk | 15:35 | |
| *** jaypipes is now known as leakypipes | 15:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add pyghmi from source non-voting test https://review.openstack.org/312607 | 15:37 |
| *** oanson has joined #openstack-infra | 15:38 | |
| *** markus_z has quit IRC | 15:38 | |
| *** jcoufal has quit IRC | 15:38 | |
| *** mikelk has quit IRC | 15:38 | |
| openstackgerrit | Chris Denneen proposed openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options https://review.openstack.org/312121 | 15:39 |
| *** dmk0202 has quit IRC | 15:39 | |
| *** nadya has quit IRC | 15:40 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 15:40 |
| *** rossella_s has quit IRC | 15:40 | |
| fungi | i've added a new 200gb ssd volume to review.o.o as a replacement for the current 100gb ssd volume. once i'm comfortable that things are still stable after the trove volume resize, i'll pvmove the extents from the old cinder volume to the new one and then extend the lv/fs to 200gb | 15:41 |
| *** rossella_s has joined #openstack-infra | 15:41 | |
| fungi | after which time i can take the old volume out of service and delete it | 15:41 |
| *** jsavak has quit IRC | 15:42 | |
| *** flepied has quit IRC | 15:42 | |
| fungi | #status log the trove instance for review.openstack.org was 10gb and 90% full, so i'm upping it to 50gb (which is supposed to be a non-impacting online operation) | 15:42 |
| openstackstatus | fungi: finished logging | 15:42 |
| fungi | #status log the current 100gb /home/gerrit2 on review.openstack.org is 95% full, so i've added a new 200gb ssd volume to review.o.o as a replacement for the current 100gb ssd volume. once i'm comfortable that things are still stable after the trove volume resize, i'll pvmove the extents from the old cinder volume to the new one and then extend the lv/fs to 200gb | 15:43 |
| openstackstatus | fungi: finished logging | 15:43 |
| *** salv-orlando has quit IRC | 15:43 | |
| *** jcoufal has joined #openstack-infra | 15:44 | |
| *** nadya has joined #openstack-infra | 15:44 | |
| anteaya | fungi: thank you | 15:46 |
| fungi | anteaya: thanks for spotting the fs growth | 15:46 |
| anteaya | welcome | 15:46 |
| *** salv-orlando has joined #openstack-infra | 15:47 | |
| fungi | we may need to bite the bullet on enabling the background git gc, but i'll bring it up again when more people are around to brainstorm | 15:47 |
| anteaya | I do hope someone has the time to dig deeper and see if this is related to the memory leak gc issue | 15:47 |
| anteaya | fungi: sounds like a good plan | 15:47 |
| *** nadya has quit IRC | 15:48 | |
| fungi | doubtful that it would be related to the jvm gc issues we're encountering, but you never know (if this is stuff egtting written temporarily to disk _and_ cached in memory) | 15:48 |
| *** nadya has joined #openstack-infra | 15:48 | |
| fungi | once we have more space on there, i'll do a normal git gc on a copy of the repo and see if i can tell what it ends up cleaning up | 15:48 |
| *** ddieterly is now known as ddieterly[away] | 15:49 | |
| AJaeger | team, it would be nice to get another +2 on the project config reviewing guidelines... Thanks! | 15:50 |
| *** shashank_hegde has joined #openstack-infra | 15:50 | |
| * AJaeger now signs off for the day | 15:50 | |
| *** nadya has quit IRC | 15:51 | |
| *** thorst_ has quit IRC | 15:51 | |
| *** yottatsa has quit IRC | 15:52 | |
| fungi | heh, rackspace has a kb article on "upgrade a cloud databases instance from mysql 5.1 to mysql 5.6" but what it describes is creating a new trove instance and then using mysqldump to copy your data out of the old instance and applying it in the new one. misleading title--got my hopes up | 15:52 |
| openstackgerrit | Anita Kuno proposed openstack-infra/infra-manual: Ensure project retirement removes dot files https://review.openstack.org/313618 | 15:53 |
| anteaya | fungi: :/ | 15:54 |
| *** apuimedo has joined #openstack-infra | 15:54 | |
| anteaya | AJaeger: enjoy your weekend | 15:54 |
| *** jsavak has joined #openstack-infra | 15:55 | |
| *** ddieterly[away] is now known as ddieterly | 15:55 | |
| apuimedo | lifeless: question about pbr. How can I do it so that after I do `python setup.py install --prefix / --root /` the installed software does not depend on git for the version | 15:55 |
| apuimedo | (on kuryr, which uses pbr) | 15:56 |
| *** lakshmiS has joined #openstack-infra | 15:56 | |
| *** davideagnello has joined #openstack-infra | 15:57 | |
| *** jlanoux has quit IRC | 15:57 | |
| *** thorst_ has joined #openstack-infra | 15:58 | |
| *** tphummel has joined #openstack-infra | 15:58 | |
| *** thorst_ has quit IRC | 16:01 | |
| anteaya | pabelanger: can you look at https://review.openstack.org/#/c/312239/ and confirm it is needed still or abandon? | 16:01 |
| *** thorst_ has joined #openstack-infra | 16:01 | |
| sigmavirus24 | Is https://storyboard.openstack.org/#!/project/760 still the place for shade bugs? | 16:02 |
| *** inc0 has quit IRC | 16:02 | |
| pabelanger | #status log OSIC leads 21 FIPs, they have been deleted manually. | 16:02 |
| openstackstatus | pabelanger: finished logging | 16:03 |
| pabelanger | anteaya: checking | 16:03 |
| anteaya | samueldmq: here is the list of folks on the project team guide team: https://review.openstack.org/#/admin/groups/953,members | 16:03 |
| sigmavirus24 | apuimedo: can you expound on that? The version *should* be generated once | 16:03 |
| *** yolanda has quit IRC | 16:03 | |
| pabelanger | anteaya: thanks, abandoned | 16:03 |
| anteaya | samueldmq: they are the folks who can approve https://review.openstack.org/#/c/264398 | 16:03 |
| anteaya | pabelanger: thank you | 16:03 |
| anteaya | sigmavirus24: yes | 16:03 |
| sigmavirus24 | Thanks anteaya :) | 16:04 |
| anteaya | sigmavirus24: welcome, thanks for asking | 16:04 |
| *** ams_ has joined #openstack-infra | 16:04 | |
| *** kjorgensen has joined #openstack-infra | 16:04 | |
| anteaya | now the follow up to that is that mordred is one of the main shade folks and as far as I know rarely reads bug reports | 16:04 |
| anteaya | but storyboard is the shade bug tracker | 16:04 |
| *** kjorgensen has quit IRC | 16:05 | |
| *** kjorgensen has joined #openstack-infra | 16:05 | |
| apuimedo | sigmavirus24: sure | 16:05 |
| apuimedo | I am making a container | 16:05 |
| pabelanger | infra-root: we might have an issue with osic-cloud1: http://paste.openstack.org/show/496353/ | 16:05 |
| apuimedo | in it, I clone the git repo | 16:05 |
| clarkb | pabelanger: sorry feel half dead this morning but if zuul was restarted the image builds in gearman would go away | 16:05 |
| apuimedo | and do python setup.py install --root / --prefix / | 16:06 |
| pabelanger | seeing a lot of connection reset by peer exceptions for osic-cloud1 | 16:06 |
| apuimedo | but when I run the project | 16:06 |
| sigmavirus24 | anteaya: Yeah I was looking for a bug related to the new betamax tests that shade is adding and wanted to contribute to the discussion :) | 16:06 |
| *** derekh has quit IRC | 16:06 | |
| apuimedo | I get the following error | 16:06 |
| clarkb | since zuul's geard us used by nodepool for image builds | 16:06 |
| pabelanger | clarkb: okay, let me confirm that. | 16:06 |
| anteaya | sigmavirus24: awesome, thank you | 16:06 |
| *** kushal has joined #openstack-infra | 16:06 | |
| zaro | morning | 16:06 |
| *** nadya has joined #openstack-infra | 16:06 | |
| anteaya | sigmavirus24: did you find anything useful yet? | 16:06 |
| *** dynarro has quit IRC | 16:06 | |
| anteaya | morning zaro | 16:06 |
| anteaya | zaro: so we have a thing | 16:06 |
| *** davidSubirosPere has quit IRC | 16:07 | |
| sigmavirus24 | anteaya: no, that's why I asked (to confirm that storyboard was still the place) | 16:07 |
| *** senk has quit IRC | 16:07 | |
| *** roxanaghe has joined #openstack-infra | 16:07 | |
| pabelanger | I cannot remember who are contact for osic is to confirm we are seeing an error | 16:07 |
| fungi | clarkb: pabelanger: yeah, if you restart zuul you also need to delete nodes in nodepool | 16:07 |
| fungi | at the moment | 16:07 |
| anteaya | zaro: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all our gerrit /home/gerrit2 is very full very fast | 16:07 |
| pabelanger | fungi: Yup, did that already | 16:07 |
| sigmavirus24 | apuimedo: go on :) (preferably using paste.openstack.org ;)) | 16:07 |
| clarkb | pabelanger: fungi should have their emails in his inbox somehwere | 16:07 |
| anteaya | zaro: it seems to have changed the rate of storage after the upgrade to 2.11 | 16:07 |
| anteaya | sigmavirus24: ah, makes sense | 16:08 |
| apuimedo | sigmavirus24: http://paste.openstack.org/show/496354/ | 16:08 |
| fungi | pabelanger: what error/behavior are you seeing? | 16:08 |
| apuimedo | sigmavirus24: yes, yes. Always paste openstack org | 16:08 |
| apuimedo | :-) | 16:08 |
| fungi | oh, you pasted it above | 16:08 |
| pabelanger | fungi: see my pastebin from nodepool | 16:08 |
| *** kzaitsev_mb has quit IRC | 16:08 | |
| anteaya | clarkb: go back to bed | 16:08 |
| pabelanger | fungi: Ya, that is one of a few execptions I am seeing | 16:08 |
| *** johnthetubaguy has quit IRC | 16:08 | |
| fungi | pabelanger: looks like fip quota exceeded probably? | 16:08 |
| *** johnthetubaguy has joined #openstack-infra | 16:08 | |
| anteaya | clarkb: get rest, drink water, consider a visit to a doctor | 16:08 |
| fungi | pabelanger: we likely leaked all the fips for all the nodes which were running there when we restarted | 16:09 |
| pabelanger | fungi: running the neutronclient commend I no longer see any leaked FIPs | 16:09 |
| pabelanger | fungi: we had 21, but I deleted them | 16:09 |
| sigmavirus24 | apuimedo: can you try using `pip install --root/ --prefix/` instead? setuptools + pbr might not be doing the right thing there with those options and I'm not sure why | 16:10 |
| *** dmellado is now known as dmellado|off | 16:10 | |
| fungi | pabelanger: but still getting that traceback in the logs? | 16:10 |
| pabelanger | fungi: yes | 16:10 |
| apuimedo | sigmavirus24: pip setup.py install? | 16:10 |
| *** psilvad has joined #openstack-infra | 16:10 | |
| pabelanger | sorry for the incoming spam | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add ubuntu-precise to bindep-fallback https://review.openstack.org/313626 | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace bare-precise with ubuntu-precise for system-config https://review.openstack.org/313627 | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise https://review.openstack.org/313628 | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Set min-ready 0 for bare-precise https://review.openstack.org/313629 | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Disable bare-precise in nodepool https://review.openstack.org/313630 | 16:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Delete bare-precise https://review.openstack.org/313631 | 16:10 |
| sigmavirus24 | apuimedo: no, just `pip install ...` | 16:10 |
| *** senk has joined #openstack-infra | 16:10 | |
| apuimedo | ok | 16:10 |
| *** flepied has joined #openstack-infra | 16:10 | |
| sigmavirus24 | apuimedo: ah yeah, you'll need `pip install --root / --prefix / .` | 16:10 |
| sigmavirus24 | You'll need the `.` to mean "install the package in this directory" | 16:11 |
| *** piet has quit IRC | 16:11 | |
| pabelanger | AJaeger: anteaya: if you have time: https://review.openstack.org/#/q/topic:ubuntu-precise is what is needed to remove bare-precise from the gate | 16:11 |
| *** roxanaghe has quit IRC | 16:12 | |
| apuimedo | sigmavirus24: does that automatically go through the requirements.txt? | 16:13 |
| * apuimedo was not familiar that pip had the setup.py install functionality built it | 16:13 | |
| apuimedo | *built in | 16:13 |
| sigmavirus24 | apuimedo: pip is magical ;) but pbr reads in requirements.txt to generate the proper data for pip to understand that it needs to install those dependencies | 16:13 |
| anteaya | pabelanger: thank you | 16:13 |
| *** apoorvad has joined #openstack-infra | 16:14 | |
| *** oanson has quit IRC | 16:14 | |
| apuimedo | sigmavirus24: http://paste.openstack.org/show/496360/ | 16:14 |
| apuimedo | sigmavirus24: is there someway for having setup.cfg mandate a oslo-config-generator run, so the installation includes the config example? | 16:15 |
| sigmavirus24 | apuimedo: what version of pip are you running? | 16:15 |
| *** sdake has joined #openstack-infra | 16:15 | |
| sigmavirus24 | and no, there's no way (I know of) to have oslo-config-generator run on install and generate a sample config | 16:16 |
| sigmavirus24 | (and I'd lobby against that anyway) | 16:16 |
| apuimedo | / # pip --version | 16:16 |
| apuimedo | pip 8.1.1 from /usr/lib/python2.7/site-packages (python 2.7) | 16:16 |
| *** woodster_ has joined #openstack-infra | 16:16 | |
| pabelanger | fungi: asking in #osic if they are away of any outages / issues | 16:16 |
| apuimedo | sigmavirus24: why would you be against it? | 16:16 |
| fungi | pabelanger: cloudnull has offered to help troubleshoot backend issues in osic in the past, so if he's around he might be able to check why our neutron create floating ip calls there are throwing keystone session errors | 16:16 |
| *** yaume has quit IRC | 16:16 | |
| apuimedo | it's nice to distribute sample conf files | 16:17 |
| sigmavirus24 | fungi: I don't think cloudnull is around this morning | 16:17 |
| *** johnny has joined #openstack-infra | 16:17 | |
| apuimedo | (with everything commented) | 16:17 |
| pabelanger | fungi: ack | 16:17 |
| *** johnny is now known as Guest34798 | 16:17 | |
| sigmavirus24 | It's a shame more #osic people don't hang out in here to start with :/ | 16:17 |
| *** yaume has joined #openstack-infra | 16:17 | |
| fungi | apuimedo: what would make more sense is to have the sample config generator run at sdist/wheel build time rather than install time | 16:17 |
| sigmavirus24 | fungi: I agree | 16:17 |
| apuimedo | fungi: yes, either of those would be fine | 16:18 |
| sigmavirus24 | apuimedo: downstream linux distributions would shoot us if we ran it at install | 16:18 |
| apuimedo | but I didn't fine a way to specify prefix on sdist | 16:18 |
| apuimedo | and I *need* to have things put in /usr/lib/docker/ forcefully | 16:18 |
| *** Nikolay_St has quit IRC | 16:18 | |
| apuimedo | not in some relative venv location | 16:19 |
| *** piet has joined #openstack-infra | 16:19 | |
| *** piet has quit IRC | 16:19 | |
| fungi | apuimedo: you may need a custom unpacker/installer of some sort? pip is designed to put things into places it considers runnable for a given environment (be it global or virtual) | 16:19 |
| apuimedo | sigmavirus24: it would look okay with archlinux package generation, I don't really see what the problem would be with rpm specfile | 16:20 |
| apuimedo | fungi: currently I use data_files | 16:20 |
| apuimedo | in pbr / setuptools to place that absolute files | 16:20 |
| apuimedo | and that works | 16:20 |
| apuimedo | but obviously I can't do it when doing sdist | 16:20 |
| apuimedo | only when I do setup.py install | 16:20 |
| *** ewilson has joined #openstack-infra | 16:21 | |
| apuimedo | sigmavirus24: any idea, about the http://paste.openstack.org/show/496360/ crash? | 16:21 |
| fungi | wow, the trove volume resize for our gerrit db is still going | 16:21 |
| sigmavirus24 | apuimedo: no I'm not sure what's going on there :( | 16:21 |
| apuimedo | :( | 16:22 |
| * sigmavirus24 hasn't tried setting up kuryr before | 16:22 | |
| apuimedo | sigmavirus24: I have a dockerfile that builds it, if you want | 16:22 |
| sigmavirus24 | pbr should know how to get the version info from a PKG-INFO file but I'm not sure if `python setup.py install` is generating that and if maybe it's having problems because of the `--root/--prefix` options your'e using | 16:23 |
| *** e0ne has quit IRC | 16:23 | |
| fungi | apuimedo: where did .../kuryr-0.1.0.dev393-py2.7.egg-info/installed-files.txt end up? | 16:23 |
| sigmavirus24 | apuimedo: might as well log a bug for it | 16:23 |
| apuimedo | http://paste.openstack.org/show/496363/ | 16:23 |
| apuimedo | let me see where it ended up | 16:23 |
| fungi | that might tell us where the effective root for installing the egg-info was | 16:24 |
| *** ruhe has quit IRC | 16:24 | |
| fungi | (which is obviously not where it then ended up looking for it) | 16:24 |
| anteaya | fungi: :/ | 16:25 |
| *** shashank_hegde has quit IRC | 16:25 | |
| *** warpc has quit IRC | 16:25 | |
| *** dprince has joined #openstack-infra | 16:25 | |
| *** dabukalam has quit IRC | 16:25 | |
| *** salv-orlando has quit IRC | 16:25 | |
| *** beisner has quit IRC | 16:25 | |
| *** dabukalam has joined #openstack-infra | 16:25 | |
| *** yamamoto has joined #openstack-infra | 16:25 | |
| *** grauzikas has quit IRC | 16:26 | |
| *** grauzikas has joined #openstack-infra | 16:26 | |
| *** warpc has joined #openstack-infra | 16:26 | |
| apuimedo | sigmavirus24: http://paste.openstack.org/show/496364/ | 16:26 |
| apuimedo | fungi: ^^ | 16:26 |
| *** vipul- has quit IRC | 16:26 | |
| *** vipuls has joined #openstack-infra | 16:27 | |
| fungi | apuimedo: got it. so it's looking for the file in a relative path, but not relative to / apparently | 16:27 |
| fungi | so cwd must be elsewhere at that point | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace bare-precise with ubuntu-precise in infra-puppet-apply-jobs https://review.openstack.org/313627 | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Delete bare-precise https://review.openstack.org/313631 | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Disable bare-precise in nodepool https://review.openstack.org/313630 | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Set min-ready 0 for bare-precise https://review.openstack.org/313629 | 16:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise https://review.openstack.org/313628 | 16:27 |
| *** SotK has quit IRC | 16:28 | |
| apuimedo | fungi: I run the uwsgi server from / | 16:28 |
| *** mrodden has quit IRC | 16:28 | |
| *** SotK has joined #openstack-infra | 16:28 | |
| fungi | yeah, that makes it all the more strange | 16:28 |
| *** devananda has quit IRC | 16:28 | |
| *** mtreinish has quit IRC | 16:28 | |
| anteaya | pabelanger: reviewed the stack, two comments on commit messages | 16:28 |
| fungi | so something is changing the working directory prior to attempting to open that file | 16:29 |
| *** thorst_ has quit IRC | 16:29 | |
| *** stevebaker has quit IRC | 16:29 | |
| *** devananda has joined #openstack-infra | 16:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Delete bare-precise https://review.openstack.org/313631 | 16:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Disable bare-precise in nodepool https://review.openstack.org/313630 | 16:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Set min-ready 0 for bare-precise https://review.openstack.org/313629 | 16:30 |
| fungi | apuimedo: oh! i misread | 16:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise https://review.openstack.org/313628 | 16:30 |
| pabelanger | anteaya: ack, updated | 16:30 |
| apuimedo | I'll put a print to see where on earth pbr tries to find it | 16:30 |
| fungi | apuimedo: so something is more likely failing to create installed-files.txt actually | 16:30 |
| *** yamamoto has quit IRC | 16:30 | |
| *** ruhe has joined #openstack-infra | 16:31 | |
| fungi | given that kuryr-0.1.0.dev393-py2.7.egg-info exists and has files under it, but not the installed-files.txt file | 16:31 |
| fungi | er | 16:31 |
| fungi | on even closer inspection, it's attempting to open for write | 16:32 |
| anteaya | pabelanger: thank you | 16:32 |
| *** yaume has quit IRC | 16:32 | |
| *** beisner has joined #openstack-infra | 16:32 | |
| *** inc0 has joined #openstack-infra | 16:32 | |
| * apuimedo running again | 16:32 | |
| fungi | so i guess that does mean it's not finding the directory (implying it's looking in the wrong place after all) | 16:32 |
| *** SergeyLukjanov has quit IRC | 16:32 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:33 | |
| *** mrodden has joined #openstack-infra | 16:33 | |
| *** thorst_ has joined #openstack-infra | 16:33 | |
| fungi | so yes, need to know what inst_files_path is at that point | 16:33 |
| *** stevebaker has joined #openstack-infra | 16:33 | |
| *** bpokorny_ has joined #openstack-infra | 16:34 | |
| *** mtreinish has joined #openstack-infra | 16:35 | |
| *** pblaho has quit IRC | 16:35 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 16:35 |
| *** jsavak has quit IRC | 16:36 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add openstack/pyghmi to the $PROJECTS variable https://review.openstack.org/313637 | 16:37 |
| apuimedo | fungi: http://paste.openstack.org/show/496367/ | 16:37 |
| *** ddieterly is now known as ddieterly[away] | 16:37 | |
| apuimedo | this is the output from the setup.py install | 16:37 |
| anteaya | jesusaur: what are your thoughts on https://review.openstack.org/#/c/193721/ | 16:38 |
| anteaya | jesusaur: it is a dusty one of yours | 16:38 |
| *** HeOS has quit IRC | 16:39 | |
| *** bpokorny has quit IRC | 16:39 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin https://review.openstack.org/312271 | 16:39 |
| jesusaur | anteaya: it depends on https://review.openstack.org/#/c/181658/ which is also dusty | 16:39 |
| anteaya | jesusaur: dusty dusty | 16:39 |
| jesusaur | I was waiting to see if Kiall would get that working, and then forgot about both | 16:39 |
| anteaya | jesusaur: what do you want to do with the two of them | 16:40 |
| anteaya | jesusaur: yeah I can understand | 16:40 |
| *** ddieterly[away] is now known as ddieterly | 16:40 | |
| Kiall | oh, theres a patch I haven't thought about it a while | 16:40 |
| *** dmk0202 has joined #openstack-infra | 16:40 | |
| fungi | apuimedo: ahh, i thought you were going to add a print(inst_files_path) before /usr/lib/python2.7/site-packages/pip/req/req_install.py line 928 and then retry to pip install the kuryr source again | 16:41 |
| * Kiall should abandon... | 16:41 | |
| *** madhu_ak has joined #openstack-infra | 16:41 | |
| *** thorst_ has quit IRC | 16:41 | |
| fungi | apuimedo: current theory is that after setup.py install completes, pip attempts to create installed-files.txt but that path is ending up not relative to / | 16:42 |
| *** sdake_ has joined #openstack-infra | 16:42 | |
| *** tosky has quit IRC | 16:43 | |
| *** nadya has quit IRC | 16:43 | |
| apuimedo | fungi: I'm now with the debugger on pbr code to see where it looks for it | 16:43 |
| apuimedo | with python setup.py installation, not pip one | 16:43 |
| fungi | ahh | 16:44 |
| *** thorst_ has joined #openstack-infra | 16:44 | |
| fungi | well, i know we've expressly said in the past that we don't support setup.py install by itself, and rely on additional magic in pip | 16:44 |
| apuimedo | fungi: really? For all of pbr? | 16:45 |
| fungi | yep, i'm looking for that section of the pbr docs now | 16:45 |
| *** tosky has joined #openstack-infra | 16:45 | |
| *** sdake has quit IRC | 16:45 | |
| fungi | oh, right there in the preamble | 16:46 |
| fungi | "Note that we don’t support the easy_install aspects of setuptools: while we depend on setup_requires, for any install_requires we recommend that they be installed prior to running setup.py install - either by hand, or by using an install tool such as pip." http://docs.openstack.org/developer/pbr/ | 16:46 |
| pabelanger | #status log osic-cloud1 is down at the moment, #osic is looking into the issue. Will update shortly. | 16:46 |
| openstackstatus | pabelanger: finished logging | 16:46 |
| *** jsavak has joined #openstack-infra | 16:46 | |
| fungi | apuimedo: so i guess setup.py install is expected to work, but only if you manually install any install_requires | 16:46 |
| anteaya | pabelanger: thanks for looking into osic | 16:47 |
| apuimedo | fungi: I always do a two step | 16:47 |
| pabelanger | anteaya: np | 16:47 |
| fungi | apuimedo: and if you use pip install instead, then you don't need to install the install_requires first | 16:47 |
| anteaya | jesusaur: whatever you and Kiall decide on that patch series is great | 16:47 |
| apuimedo | first pip install -r requirements :P | 16:47 |
| apuimedo | fungi: truw | 16:47 |
| apuimedo | *true | 16:47 |
| anteaya | I have to go for a walk now, apparently one of my neighbours wants something | 16:48 |
| anteaya | back in a while | 16:48 |
| *** thorst_ has quit IRC | 16:48 | |
| *** piet has joined #openstack-infra | 16:49 | |
| *** dimtruck is now known as zz_dimtruck | 16:49 | |
| *** electrofelix has quit IRC | 16:50 | |
| *** thorst_ has joined #openstack-infra | 16:50 | |
| fungi | apuimedo: so i'm probably in need of a reqind to the original problem statement. you were saying something about needing to install a particular file into /usr/lib/docker, and i do see there is says "copying etc/kuryr.spec -> /usr/lib/docker/plugins/kuryr" | 16:50 |
| *** burgerk_ has quit IRC | 16:50 | |
| fungi | s/reqind/rewind/ | 16:50 |
| apuimedo | yes, that works, because I use the --prefix / --root /, if I use sdist or bdist (which would probably fix the missing pbr version) I can't have that file, since those don't have prefix nor root | 16:52 |
| *** gyee has joined #openstack-infra | 16:52 | |
| pabelanger | success, #osic was able to fix the issue: http://grafana.openstack.org/dashboard/db/nodepool-osic | 16:52 |
| pabelanger | they are still debugging the problem however | 16:53 |
| zaro | anteaya: ohh wow, yeah. that looks bad | 16:54 |
| fungi | anteaya: looks like your friends in the scientific working group have reiterated their question, this time to the infra ml, on whether they can use storyboard? | 16:54 |
| *** burgerk has joined #openstack-infra | 16:54 | |
| zaro | anteaya: probably can be temporarily relieved by running git gc | 16:54 |
| *** flepied has quit IRC | 16:55 | |
| *** thorst_ has quit IRC | 16:55 | |
| *** sridhar_ram has joined #openstack-infra | 16:55 | |
| zaro | fungi: anteaya pointed out that we have pretty much run out of disk space on gerrit (or on /home/gerrit2 volume). | 16:55 |
| *** dmk0202 has quit IRC | 16:55 | |
| fungi | zaro: yes, there was much subsequent discussion | 16:56 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting https://review.openstack.org/312596 | 16:56 |
| * zaro checks | 16:56 | |
| *** thorst_ has joined #openstack-infra | 16:56 | |
| fungi | zaro: we're also almost out of space in the db, so i'm attempting to rectify that first and then i'll fix the server filesystem space issue | 16:56 |
| pabelanger | #status log osic-cloud1 is coming back online. Thanks for the help #osic | 16:56 |
| openstackstatus | pabelanger: finished logging | 16:56 |
| *** roxanaghe has joined #openstack-infra | 16:57 | |
| fungi | pabelanger: what was the issue, ultimately? | 16:57 |
| *** dmk0202 has joined #openstack-infra | 16:57 | |
| pabelanger | fungi: they made some changes to their F5 load balancer it looks like. So reverting the changes fixed our issue. | 16:57 |
| pabelanger | fungi: they are still hunting for the root cause | 16:58 |
| *** rhochmuth has quit IRC | 16:58 | |
| fungi | got it. thanks! | 16:58 |
| pabelanger | fungi: They are discussing the issue in #osic now | 16:58 |
| zaro | fungi, anteaya : i was planning to verify gerrit's git gc but didn't have time this week. i'll put move it to the top of my priority for next week. hopefully we can start running git gc to reduce rate of disk consumption | 16:59 |
| *** yamamoto has joined #openstack-infra | 16:59 | |
| fungi | apuimedo: so where is the missing pbr version problem illustrated when using setup.py install? i see it managed to figure out this was kuryr 0.1.0.dev393 when creating the egg-info | 16:59 |
| *** jerryz has joined #openstack-infra | 16:59 | |
| *** sridhar_ram has quit IRC | 16:59 | |
| *** sridhar_ram has joined #openstack-infra | 17:00 | |
| apuimedo | it happens when I try to run the software, as it likes to print its version | 17:00 |
| *** thorst_ has quit IRC | 17:01 | |
| fungi | apuimedo: got it. so starting the kuryr daemon, pbr.version throws an exception because it can't find the pbr.json file? | 17:01 |
| *** dmk0202 has quit IRC | 17:01 | |
| *** shardy has quit IRC | 17:02 | |
| apuimedo | yup | 17:02 |
| mordred | mmm. this sounds like a fun bug | 17:02 |
| *** burgerk has quit IRC | 17:02 | |
| *** thorst_ has joined #openstack-infra | 17:02 | |
| fungi | does the python system path include /lib/python2.7/site-packages? | 17:02 |
| *** flepied has joined #openstack-infra | 17:05 | |
| *** roxanaghe has quit IRC | 17:06 | |
| *** yamamoto has quit IRC | 17:06 | |
| *** kushal has quit IRC | 17:07 | |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 17:07 |
| *** jistr has quit IRC | 17:07 | |
| *** thorst_ has quit IRC | 17:07 | |
| *** thorst_ has joined #openstack-infra | 17:08 | |
| *** apuimedo has quit IRC | 17:09 | |
| *** mestery has quit IRC | 17:10 | |
| *** mugsie has quit IRC | 17:10 | |
| *** yamahata has joined #openstack-infra | 17:12 | |
| *** roxanaghe has joined #openstack-infra | 17:12 | |
| *** itsuugo has quit IRC | 17:13 | |
| *** kushal has joined #openstack-infra | 17:13 | |
| *** diana_clarke has left #openstack-infra | 17:13 | |
| *** itsuugo has joined #openstack-infra | 17:15 | |
| *** jsavak has quit IRC | 17:16 | |
| *** bpokorny_ has quit IRC | 17:16 | |
| *** thorst_ has quit IRC | 17:17 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Calculate deployment times for the Heat resources https://review.openstack.org/313097 | 17:17 |
| *** uberjay_ has quit IRC | 17:18 | |
| *** weshay has quit IRC | 17:18 | |
| *** thorst_ has joined #openstack-infra | 17:19 | |
| *** ddieterly is now known as ddieterly[away] | 17:19 | |
| JayF | https://review.openstack.org/#/c/313637 is a project-config change, one line change, would love to get a quick review and merge if possible to fix a new job we created | 17:20 |
| *** uberjay has joined #openstack-infra | 17:20 | |
| *** matt-borland has quit IRC | 17:20 | |
| *** jtomasek has joined #openstack-infra | 17:21 | |
| *** lezbar has quit IRC | 17:22 | |
| *** thorst_ has quit IRC | 17:23 | |
| *** flepied has quit IRC | 17:23 | |
| *** flepied has joined #openstack-infra | 17:25 | |
| *** thorst_ has joined #openstack-infra | 17:25 | |
| *** uberjay has quit IRC | 17:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Initial support for centos-7 jenkins slave https://review.openstack.org/312725 | 17:28 |
| *** Jeffrey4l has quit IRC | 17:28 | |
| *** roxanaghe has quit IRC | 17:29 | |
| *** roxanaghe has joined #openstack-infra | 17:29 | |
| *** Nikolay_St has joined #openstack-infra | 17:29 | |
| *** inc0 has quit IRC | 17:29 | |
| anteaya | zaro: thank you | 17:30 |
| *** thorst_ has quit IRC | 17:30 | |
| anteaya | fungi: yes I asked the scientific working group to go public with the question | 17:30 |
| anteaya | fungi: so same question, better venue, they are learning | 17:31 |
| *** ddieterly[away] is now known as ddieterly | 17:32 | |
| fungi | excellent | 17:32 |
| *** thorst_ has joined #openstack-infra | 17:33 | |
| *** shashank_hegde has joined #openstack-infra | 17:33 | |
| *** flepied has quit IRC | 17:34 | |
| anteaya | they are a great group | 17:34 |
| anteaya | finding their way | 17:34 |
| *** tojuvone has joined #openstack-infra | 17:35 | |
| *** mestery has joined #openstack-infra | 17:37 | |
| *** apuimedo has joined #openstack-infra | 17:38 | |
| *** thorst_ has quit IRC | 17:38 | |
| pabelanger | fungi: AJaeger: project-config-core: do you mind taking a moment to review https://review.openstack.org/#/q/topic:ubuntu-precise our stack to migrate away from bare-precise | 17:38 |
| pabelanger | happy to report testing ubuntu-precise worked out well! | 17:38 |
| *** thorst_ has joined #openstack-infra | 17:39 | |
| *** jsavak has joined #openstack-infra | 17:39 | |
| *** mugsie has joined #openstack-infra | 17:41 | |
| *** jang has quit IRC | 17:41 | |
| *** jsavak has quit IRC | 17:45 | |
| *** ddieterly is now known as ddieterly[away] | 17:45 | |
| *** yolanda has joined #openstack-infra | 17:46 | |
| *** jsavak has joined #openstack-infra | 17:46 | |
| *** thorst_ has quit IRC | 17:48 | |
| prometheanfire | SamYaple: I'm testing your two glean patches now | 17:48 |
| prometheanfire | well, building the image | 17:49 |
| *** thorst_ has joined #openstack-infra | 17:49 | |
| *** uberjay has joined #openstack-infra | 17:50 | |
| openstackgerrit | Merged openstack-infra/system-config: Minor documentation tweaks https://review.openstack.org/310637 | 17:52 |
| *** sambetts is now known as sambetts|afk | 17:53 | |
| *** yolanda has quit IRC | 17:54 | |
| *** ddieterly[away] is now known as ddieterly | 17:54 | |
| *** thorst_ has quit IRC | 17:55 | |
| *** thorst_ has joined #openstack-infra | 17:56 | |
| *** yarkot1 has quit IRC | 17:57 | |
| *** sdake_ is now known as sdake | 17:57 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Swift functional tests in Identity v3-only https://review.openstack.org/313659 | 17:58 |
| *** thorst_ has quit IRC | 18:00 | |
| *** sridhar_ram has quit IRC | 18:00 | |
| *** thorst_ has joined #openstack-infra | 18:02 | |
| *** weshay has joined #openstack-infra | 18:02 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: tests/elements/fake-os: add '/tmp' as top level dir https://review.openstack.org/313660 | 18:04 |
| *** ddieterly is now known as ddieterly[away] | 18:04 | |
| *** thorst_ has quit IRC | 18:06 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:06 | |
| EmilienM | is it possible for infra to delete 8.0.0 branch on openstack/puppet-aodh? I think it was created by mistake. Thanks | 18:07 |
| fungi | EmilienM: sure, just a sec | 18:07 |
| *** yarkot1 has joined #openstack-infra | 18:07 | |
| *** lucasagomes is now known as lucas-afk | 18:07 | |
| *** tosky has quit IRC | 18:08 | |
| *** thorst_ has joined #openstack-infra | 18:08 | |
| krotscheck | fungi: Did you get around to running that permissions script? If not, would you mind? | 18:08 |
| EmilienM | thanks! | 18:08 |
| fungi | krotscheck: did it yesterday or the day before | 18:08 |
| krotscheck | fungi: Awesome, thank you. | 18:08 |
| fungi | krotscheck: let me know if gerrit isn't acting how you expect | 18:09 |
| *** bpokorny has joined #openstack-infra | 18:09 | |
| *** matt-borland has joined #openstack-infra | 18:10 | |
| Kiall | Heya, any requirements-core's about for a trivial projects.txt addition? https://review.openstack.org/#/c/309127/ | 18:11 |
| Kiall | and project-config core's for https://review.openstack.org/#/c/313530/ ? Thanks :) | 18:11 |
| *** thorst_ has quit IRC | 18:12 | |
| fungi | EmilienM: i've deleted the 8.0.0 branch from openstack/puppet-aodh (its head was 3f0e8910b2703ed7f23ec22a8ec86e575c5f922c) | 18:14 |
| *** thorst_ has joined #openstack-infra | 18:14 | |
| EmilienM | fungi: thank you Sir. Someone (if not me) created it by mistake. | 18:14 |
| *** mriedem1 has joined #openstack-infra | 18:15 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Change urllib3 version to urllib3>=1.15 https://review.openstack.org/312172 | 18:15 |
| *** mriedem has quit IRC | 18:17 | |
| *** thorst_ has quit IRC | 18:18 | |
| *** lhinds has quit IRC | 18:19 | |
| *** mriedem1 is now known as mriedem | 18:20 | |
| *** thorst_ has joined #openstack-infra | 18:20 | |
| *** tonytan4ever has quit IRC | 18:22 | |
| *** BigWillie has quit IRC | 18:24 | |
| *** salv-orlando has joined #openstack-infra | 18:24 | |
| *** inc0 has joined #openstack-infra | 18:25 | |
| *** BigWillie has joined #openstack-infra | 18:26 | |
| *** nadya has joined #openstack-infra | 18:27 | |
| * mordred now has driver's license | 18:29 | |
| * mordred would like to let everyone know that the tx driver's license stuff is AMAZINGLY well done | 18:30 | |
| bkero | How so? | 18:30 |
| mordred | as in, you can get in line ahead of time remotely and they text you ETAs and then when you're up | 18:30 |
| *** sdake_ has joined #openstack-infra | 18:31 | |
| fungi | the fake cattle crossing on the test course are reportedly very lifelike | 18:31 |
| * crinkle hopes mordred had some form of drivers license before now | 18:31 | |
| mordred | so there is no "stand in line for 2 hours" part of the process | 18:31 |
| clarkb | what does a horseback rider with a raised arm mean? | 18:31 |
| mordred | crinkle: it's a question you should not ask in too much depth ;) | 18:31 |
| greghaynes | hahaha | 18:31 |
| clarkb | (real OR DL test question) | 18:31 |
| crinkle | mordred: this is concerning | 18:31 |
| bkero | mordred: I'm wondering how the rental agency gave you a van without license | 18:31 |
| * mordred may have still had a valid WA driver's license | 18:32 | |
| mordred | you konw - from having lived there years ago | 18:32 |
| *** sridhar_ram has joined #openstack-infra | 18:32 | |
| * mordred may not have gotten his license changed in ny, la, ny, ca, or ny | 18:33 | |
| *** sdake has quit IRC | 18:33 | |
| clarkb | mordred: in vancouver across the river they do first week of school stings | 18:34 |
| fungi | perhaps some day we'll have the option of a national driving license | 18:34 |
| clarkb | drop your kid off with oregon plates, thats a paddling | 18:34 |
| *** yottatsa has joined #openstack-infra | 18:35 | |
| *** nadya has quit IRC | 18:35 | |
| mordred | fungi: I'm actually a fan of the state licenses - or, at least I was until they all started sharing information with each other | 18:35 |
| *** yottatsa has quit IRC | 18:35 | |
| fungi | touché | 18:35 |
| mordred | made it easier to just ignore silly tickets from the chesapeake bay bridge | 18:35 |
| SpamapS | mordred: CA has the same process for DMV btw, except they don't text you, you just make an appointment and show up and they see you within 5 minutes. | 18:35 |
| mordred | SpamapS: it's so much more civilized than when I was 16 | 18:36 |
| JayF | mordred: in cali they do appointments, basically you wait ~30m if you have an appointment, all day otherwise | 18:36 |
| SpamapS | yeah it's not a hell hole anymore | 18:36 |
| *** gordc has quit IRC | 18:36 | |
| SpamapS | JayF: err, your DMV office sucks. ;) | 18:36 |
| SpamapS | I've never waited more than 5 minutes with an appointment. | 18:36 |
| JayF | SpamapS: All 3 I've been to (SF Fell St, Daly City, San Jose) have been that way | 18:36 |
| SpamapS | Even at the Hollywood DMV where there is a line of cars and people around the block every day. | 18:36 |
| mordred | in any case, for the first time since before the openstack project started, my driver's license actually matches my residence | 18:36 |
| SpamapS | mordred: be afraid | 18:37 |
| fungi | cold toilet seats, dentist chairs and trips to dmv... call it pointless | 18:37 |
| mordred | SpamapS: it's weird man | 18:37 |
| * JayF hopes WA DMV is not too bad, as I'll have to jump through those hoops in 2 months | 18:37 | |
| JayF | and I think it's even more painful now that my car is paid for, b/c I'll have to move the title too | 18:37 |
| bkero | OR DMV is pretty bad. It's all classic take-a-number-and-sit-down. | 18:37 |
| mordred | JayF: I never had problems in WA - the downtown seattle office is quite small and has not a ton of people there the times I went | 18:37 |
| docaedo | bkero: interesting I was just going to say my OR DMV experiences have been great. But maybe less crowded in the 'burbs | 18:37 |
| clarkb | jayf WA is good | 18:37 |
| fungi | nc was pretty bad the last time i had to go, but that's been nearly a decade. hopefully its improved | 18:38 |
| clarkb | JayF title and new license was super simple | 18:38 |
| bkero | docaedo: which one do you go to? | 18:38 |
| clarkb | thry just accept you can drive if ypu have another state dl | 18:38 |
| docaedo | bkero: Tualatin | 18:38 |
| anteaya | what is this waiting you are talking about? | 18:38 |
| leifmadsen | pabelanger: thanks for merging my doc changes :) | 18:38 |
| *** piet has quit IRC | 18:38 | |
| anteaya | lineup? what is that? | 18:38 |
| pabelanger | leifmadsen: np | 18:38 |
| clarkb | jayf they take your photo, give you a temp and done | 18:39 |
| JayF | clarkb: sweet, thanks :). | 18:39 |
| JayF | clarkb: you in seattle metro area? | 18:39 |
| clarkb | JayF I was but moved back to portland a year and a half ago | 18:39 |
| JayF | ah | 18:39 |
| *** e0ne has joined #openstack-infra | 18:39 | |
| bkero | docaedo: sunset tc dmv was pretty terrible | 18:39 |
| clarkb | bkero its not bad early in the day | 18:39 |
| clarkb | that is where I got the horseback rider question | 18:40 |
| *** savihou has quit IRC | 18:40 | |
| bkero | haha | 18:40 |
| docaedo | clarkb: is the answer "the rider is saying howdy!" ? | 18:41 |
| clarkb | docaedo: its something like riders ahead. I forget | 18:42 |
| clarkb | the worst dmv is over by gateway | 18:42 |
| clarkb | that is where I got my forst murican dl | 18:42 |
| clarkb | took several days of back and forth nonsense | 18:42 |
| greghaynes | clarkb: docaedo its always that youre scaring the horse | 18:42 |
| greghaynes | guess who just got a license a few months ago | 18:43 |
| clarkb | at first they said my passport and micronesian dl were good then they said no I had to do all the testing and I had to go to county records office to get birth certifi ate | 18:43 |
| clarkb | I was not happy with them and do not intend on ever returning | 18:43 |
| *** ociuhandu has quit IRC | 18:43 | |
| mordred | greghaynes: that's actually the real answer? | 18:43 |
| mordred | clarkb: no. you should not | 18:43 |
| greghaynes | mordred: yes | 18:43 |
| *** dynarro has joined #openstack-infra | 18:44 | |
| greghaynes | mordred: in portland we are very concerned about scaring horses | 18:44 |
| docaedo | greghaynes: fair enough. Have ridden horses and being on a scared one is no fun. | 18:44 |
| clarkb | also in 2018 it will be useless for travel | 18:44 |
| clarkb | WA OR and a bunch of other states wont meet federal id requirements | 18:45 |
| bkero | Because we're not doing the RealID thing? | 18:45 |
| clarkb | yes | 18:45 |
| clarkb | so get a passport or pay more for real id | 18:45 |
| bkero | wfm, I have my GlobalEntry ID I can use anyways | 18:46 |
| amrith | is anyone really doing that realID stuff or will it all fall apart? | 18:46 |
| amrith | can't drive across state lines | 18:46 |
| bkero | clarkb: do globalentry, get one of these https://www.united.com/web/format/img/travel/how_to_use_a_cbp_trusted_traveler_pass_id_to_access_tpc.jpg?01AD=3SjrxFUokQGaO_Tfoicg2bKIEqx3-HmKrurRhJ21Roczbb77MiNWk7A&01RI=1E5D92D4FF64A87&01NA=na | 18:46 |
| bkero | Even people in the airports haven't seen them before | 18:46 |
| bkero | but it's technically a federally-issued ID | 18:46 |
| pabelanger | my passport has RFID.... what could possible go wrong | 18:47 |
| clarkb | amrith an or dl is still fine for driving elsewhere just not for tsa to know who you are | 18:47 |
| amrith | that's interesting because I've started using my passport card for tsa as a matter of course | 18:47 |
| fungi | yeah, i use my usa passport card as photo id, unless i actually have to prove i'm licensed to operate a motor vehicle | 18:47 |
| mordred | bkero: that's an awesome idea. I should totally start doing that just to screw with the tsas | 18:47 |
| amrith | I've been had no trouble with that, but some building security folks don't know what a passport card is and won't accept it. | 18:47 |
| bkero | mordred: Haha, let me know how it works out for you :) | 18:47 |
| fungi | amrith: somehow i've not had any trouble there. i also use it when purchasing alcoholic beverages | 18:48 |
| anteaya | what is a passport card? | 18:48 |
| amrith | mordred, remember that 'screwing with tsas' is on the list of famous last words like 'look honey, no hands', and 'i wonder whether this mushroom is toxic' | 18:49 |
| bkero | "Hold my beer...watch this!" | 18:49 |
| fungi | anteaya: the usa passport office issues them as an alternative to our passport book, though it's only good for international travel by land or water to canada, mexico and the caribbean | 18:49 |
| anteaya | fungi: ah, interesting | 18:49 |
| fungi | but it's basically the size of a credit card, so easier to carry around | 18:50 |
| anteaya | are they less expensive? do they have a longer expiration date? | 18:50 |
| * anteaya is trying to see the appeal | 18:50 | |
| anteaya | oh okay | 18:50 |
| amrith | anteaya, it is easy to carry around | 18:50 |
| anteaya | wow | 18:50 |
| fungi | yeah they're also slightly lower cost, though that doens't matter much for me since i still need the full passport book too | 18:50 |
| anteaya | right | 18:50 |
| amrith | https://travel.state.gov/content/passports/en/passports/information/card.html | 18:50 |
| anteaya | we have an enhanced driver's licence | 18:50 |
| bkero | I figured the cards were only useful for people who lived along the border and had to cross often. Easier/safer than passport. | 18:51 |
| anteaya | which allows you land access to the states | 18:51 |
| anteaya | it appeals to drivers from canada with no desire to fly | 18:51 |
| fungi | bkero: also convenient if you go on cruises or otherwise travel to islands in the caribbean | 18:52 |
| bkero | fungi: good point, I think the caribbean thing is new | 18:52 |
| * bkero been meaning to go on a focus-working cruise | 18:52 | |
| fungi | or if you live in a usa protectorate in the caribbean | 18:52 |
| fungi | since island hopping is pretty easy there | 18:53 |
| bkero | true | 18:53 |
| bkero | sounds like fun :) | 18:53 |
| fungi | my wife and i have talked about relocating there at some point. st. croix is pretty cheap actually | 18:54 |
| docaedo | bkero: you should join the Community App Catalog team, our mid-cycle is either going to be in Hawaii, or on a caribbean cruise | 18:54 |
| fungi | cruise mid-cycles seem like they could work out well | 18:55 |
| fungi | as long as you can line up a conference room on the ship | 18:55 |
| * docaedo is kidding but hopes the idea will catch on just the same ;) | 18:55 | |
| fungi | finding places for a group to focus can be a little challenging otherwise | 18:55 |
| crinkle | lock everyone on a boat until the work is done, sounds productive | 18:56 |
| pabelanger | boattboaty mcboatface here we come | 18:56 |
| fungi | we regularly go cruising with a large group of friends, and when a dozen or so of us want to find a quiet spot to have a card game or something it can be tough sometimes | 18:56 |
| pabelanger | err boaty* | 18:56 |
| nibalizer | pabelanger: cloudy mccloudface | 18:56 |
| pabelanger | nibalizer: YES | 18:57 |
| bkero | docaedo: sounds awesome :) Where do I start? | 18:57 |
| fungi | nibalizer: excellent alternative to "vanilla" | 18:57 |
| nibalizer | oh god | 18:57 |
| nibalizer | this is why we can't trust democracy | 18:57 |
| bkero | fungi: do cruise ships have conference rooms? | 18:57 |
| fungi | i have to assume so | 18:57 |
| docaedo | Lots of rooms but little bandwidth | 18:57 |
| fungi | though it's a very good question | 18:57 |
| fungi | you can privately book lounges for a group | 18:57 |
| bkero | docaedo: sounds perfect for a meeting | 18:57 |
| fungi | but hard to get one all day | 18:58 |
| clarkb | thry didnt name the ship that | 18:58 |
| fungi | and yes, internet access is not great (though it's still way better than it used to be) | 18:58 |
| clarkb | just the submersible | 18:58 |
| anteaya | pabelanger: it is a sub | 18:58 |
| bkero | That's the first I'm hearing of internet access that has been more available than <1mbit for $5/min | 18:58 |
| fungi | clarkb: i've actually convinced christine to let me name our boat that once we get around to picking out what we want | 18:58 |
| pabelanger | anybody feel like +A a few ubuntu-precise changes? https://review.openstack.org/#/q/topic:ubuntu-precise | 18:59 |
| pabelanger | anteaya: Ya, I see that | 18:59 |
| clarkb | Iama fan of bad pun boat names | 19:00 |
| fungi | the Review-MySQL trove instance in rax-dfw claims it is _still_ resizing its volume. i've opened a support ticket | 19:00 |
| fungi | need to accept the possibility that we might have to take a gerrit outage to mysqldump and then load into a new trove instance | 19:02 |
| *** javeriak has quit IRC | 19:02 | |
| pabelanger | fungi: ack | 19:02 |
| anteaya | fungi: ack | 19:02 |
| anteaya | I'm going for a walk | 19:02 |
| anteaya | happy new moon in Taurus | 19:02 |
| anteaya | back in a while | 19:02 |
| openstackgerrit | Merged openstack-infra/system-config: Use double quotes around bash variables https://review.openstack.org/311635 | 19:03 |
| *** dynarro has quit IRC | 19:03 | |
| fungi | once ci activity drops off a bit as the day goes on, i'm going to go ahead and start the pvmove for /home/gerrit2 onto the larger pv (i'll do it in a root screen session so other infra-root can attach and check on it in case it runs for longer than i expect) | 19:04 |
| *** ddieterly[away] has quit IRC | 19:04 | |
| fungi | it'll cause some filesystem performance degradation, so don't want to start it while people are heavily using the server | 19:04 |
| *** Nikolay_St has quit IRC | 19:04 | |
| mordred | fungi: ++ | 19:05 |
| *** dynarro has joined #openstack-infra | 19:05 | |
| nibalizer | fungi: rgr | 19:05 |
| fungi | #status log opened support ticket 160506-iad-0001201 for Review-MySQL trove instance taking >3 hours (so far) to resize its backing volume | 19:06 |
| openstackstatus | fungi: finished logging | 19:06 |
| *** asselin has joined #openstack-infra | 19:07 | |
| *** tonytan4ever has joined #openstack-infra | 19:07 | |
| *** ifarkas has quit IRC | 19:08 | |
| amrith | fungi, I'm not sure how rax runs their db service but can you see the cinder volume in cinder list? | 19:08 |
| amrith | or are they running the nova instances and cinder volumes in some shadow tenant? | 19:09 |
| nibalizer | so just to confirm, looks like we're doing local mysql for storyboard? | 19:09 |
| nibalizer | not trove | 19:09 |
| fungi | amrith: i've never seen them show up in cinder list | 19:09 |
| nibalizer | er nvm | 19:10 |
| nibalizer | the other way around | 19:10 |
| mordred | nibalizer: no - we should be doing | 19:10 |
| mordred | yah | 19:10 |
| mordred | phew | 19:10 |
| *** ewilson has quit IRC | 19:10 | |
| mordred | that was almost a problem | 19:10 |
| amrith | fungi, then they're operating it with a shadow tenant. good ... | 19:10 |
| fungi | nibalizer: yes, the current storyboard production db is in a trove instance | 19:10 |
| *** javeriak has joined #openstack-infra | 19:10 | |
| nibalizer | ok, so the dev server can just have a dev trove yah? | 19:11 |
| mordred | amrith: while we're talking to you about trove - one of the things we'd love to see is ingress and egress replication | 19:11 |
| mordred | amrith: at different times we've needed both ... | 19:11 |
| *** e0ne has quit IRC | 19:11 | |
| mordred | amrith: ingress as in "I have a mysql server in a nova instance but I want to migrate to using a trove instance, so I'd like to set up the trove instance as a replication slave of my current instance and then do a short failover" | 19:12 |
| amrith | mordred, replication on the data injest side you mean? | 19:12 |
| fungi | nibalizer: there's already a StoryBoard-Dev-MySQL trove instance in dfw for the openstackci tenant. pretty sure i created that in preparation for the server which never materialized | 19:12 |
| *** ddieterly has joined #openstack-infra | 19:12 | |
| amrith | YEAH! | 19:12 |
| amrith | that would be awesome | 19:12 |
| *** ociuhandu has joined #openstack-infra | 19:12 | |
| fungi | also continuous replication from trove in one provider/region to trove in another | 19:12 |
| mordred | amrith: egress as in "I want to run another mysql $somewhere not in this trove and replicate to it - perhaps for reporting or backups or whatever" | 19:12 |
| fungi | yeah, that | 19:12 |
| nibalizer | oh neat | 19:13 |
| amrith | mordred, will log those ... | 19:13 |
| fungi | which could also be trove as teh source, or just mysql in a nova instance or on a real server somewhere or... | 19:13 |
| nibalizer | fungi: well there is a review up to create a dev server | 19:13 |
| mordred | amrith: and finally - I've love to be able to do the "run my schema upgrade on a slave, then promote the slave to master" all inside of trove | 19:13 |
| mordred | amrith: and/or "do a schema upgrade on one node of a galera cluster at a time" | 19:13 |
| nibalizer | so one day we might even make the server | 19:13 |
| amrith | mordred, I think that works now with standard replication | 19:13 |
| amrith | as in mysql replication | 19:13 |
| *** flwang1 has joined #openstack-infra | 19:14 | |
| amrith | I'm not sure that this is a valid thing with galera because with galera I think it'll replicate the upgrade, no? | 19:14 |
| mordred | there are two different galera settings | 19:14 |
| mordred | one is total order isolation which is when you want to make sure the upgrade happens everywhere | 19:14 |
| amrith | you speak of RSU | 19:14 |
| fungi | nibalizer: yep! just saying it was a half-completed effort, so there are some things (like the trove instance) already done. i probably also added it to hiera, but if so then it likely got lost again when we split up the hiera file since keys not referenced in the global site manifest weren't included in the new split hiera files | 19:14 |
| amrith | I guess | 19:14 |
| mordred | there's also RSU | 19:14 |
| mordred | yah | 19:14 |
| amrith | TOI is the other | 19:15 |
| mordred | exactly | 19:15 |
| amrith | so trove now supports galera | 19:15 |
| amrith | use the pxc guest image | 19:15 |
| mordred | cool. I will investigate doing this in our current account | 19:15 |
| amrith | iirc you can change modes with something like | 19:15 |
| fungi | nibalizer: though you may be able to git log -p in /etc/puppet/hieradata and keyword search for it | 19:16 |
| amrith | set wsrep_osu_meth = 'RSU' | 19:16 |
| amrith | or some such thing | 19:16 |
| amrith | my galera knowledge is a bit rusty | 19:16 |
| amrith | once you do that if you get the nodes to all go to RSU, I think it should work. | 19:16 |
| amrith | I wrote that down on my lsit of things to try | 19:16 |
| amrith | will get back to you | 19:16 |
| *** gordc has joined #openstack-infra | 19:17 | |
| *** ewilson has joined #openstack-infra | 19:18 | |
| *** roxanaghe has quit IRC | 19:18 | |
| fungi | amrith: yeah, that's probably our biggest headache with using trove at this point. we want cross-provider backups of our data, and currently do it in a very ugly and performance-impacting fashion by running a full mysqldump on a nova instance and then performing a network copy to another nova instance somewhere else | 19:20 |
| fungi | so every day at 00:00 utc, gerrit and etherpad and other things become unresponsive for several minutes | 19:21 |
| fungi | and if we could continuously replicate to a read-only trove instance elsewhere instead, we could just take scheduled snapshots of that or something | 19:22 |
| amrith | fungi, where's your trove instance physically running? rax? | 19:22 |
| fungi | (or pause replication and take a mysqldump of the read-only replica or whatever) | 19:22 |
| fungi | amrith: yeah, rackspace dfw right now | 19:22 |
| *** madhu_ak has quit IRC | 19:22 | |
| *** cody-somerville has joined #openstack-infra | 19:22 | |
| amrith | so I know this isn't ideal, but here's a thought | 19:23 |
| amrith | can you setup a second trove instance somewhere and setup replication manually? | 19:23 |
| amrith | after all, everything you need to setup replication goes over the standard mysql ports. | 19:23 |
| amrith | and you can set it up with the SQL interface. | 19:24 |
| fungi | amrith: oh, really? that's a thought | 19:24 |
| amrith | I'd start with a painful mysqldump with a transaction id | 19:24 |
| *** flwang1 has quit IRC | 19:24 | |
| amrith | fungi, it is a cludge | 19:24 |
| amrith | or is that kludge? | 19:24 |
| fungi | also sometimes kluge | 19:24 |
| amrith | that thing :) | 19:24 |
| *** sdake has joined #openstack-infra | 19:24 | |
| *** ewilson has quit IRC | 19:25 | |
| fungi | loan words are fun | 19:25 |
| *** sdake has quit IRC | 19:25 | |
| amrith | we're working on snapshot replication in newton | 19:26 |
| *** rcernin has joined #openstack-infra | 19:26 | |
| *** ewilson has joined #openstack-infra | 19:26 | |
| amrith | you'll be able to take a cinder snapshot | 19:26 |
| amrith | and use that as a backup ... | 19:26 |
| *** flwang1 has joined #openstack-infra | 19:26 | |
| *** sdake_ has quit IRC | 19:26 | |
| mordred | amrith: so - I thought last time I looked in to doing that | 19:28 |
| mordred | there was something I couldn't do | 19:28 |
| mordred | amrith: but I think the problem was that was before trove supported replication at all | 19:28 |
| fungi | amrith: when you say setup replication manually, you mean replication can be configured via the sql interface, or you mean "replicate" by taking a periodic (incremental?) mysqldump of the source and then push that into the destination as an update? | 19:28 |
| mordred | so my problem was getting binlogging turned on | 19:28 |
| amrith | yes, you'll need logging turned on | 19:29 |
| mordred | fungi: no - the dump is a thing you do to get the slave started | 19:29 |
| amrith | so here's how you get around that ... | 19:29 |
| mordred | fungi: if you dump with the replicatoin info included | 19:29 |
| fungi | aha, now i understand | 19:29 |
| amrith | well, you are in RAX | 19:29 |
| fungi | so just bootstrap via a mysqldump with a transaction id so it knows where to continue from | 19:29 |
| mordred | yup | 19:29 |
| amrith | their trove is slightly different from upstream | 19:29 |
| mordred | amrith: of course it is | 19:29 |
| amrith | no, I mean forked different | 19:30 |
| mordred | yah | 19:30 |
| amrith | and maybe not up to date with upstream | 19:30 |
| amrith | so assume ... | 19:30 |
| mordred | I can think of literally zero parts of openstack that that is not the case with there | 19:30 |
| amrith | that you went to (say bb) | 19:30 |
| fungi | amrith: wait, are you suggesting that rackspace might run non-upstream versions of openstack services? | 19:30 |
| amrith | you could launch instance 1 | 19:30 |
| amrith | then launch instance 2 replica of instance 1 | 19:30 |
| amrith | that causes binlogging to get turned on | 19:30 |
| mordred | yup | 19:30 |
| amrith | now in some other cloud far away, launch a trove | 19:30 |
| amrith | and manually configure it to be a slave | 19:30 |
| amrith | won't that work? | 19:30 |
| amrith | I don't know what would happen in rax, haven't used their db service in a while. | 19:31 |
| amrith | mordred, let me try this out and let you know | 19:31 |
| amrith | easy enough for me to spin up trove instances | 19:31 |
| amrith | oh, wait. would you know how I have to build guest images for trove? :> | 19:32 |
| mordred | amrith: hahah | 19:32 |
| mordred | amrith: it's ok - that's great info and enough for me to go poking | 19:32 |
| amrith | np, got me curious now. am just launching a couple ... | 19:33 |
| mordred | :) | 19:33 |
| *** yonglihe has quit IRC | 19:33 | |
| *** madhu_ak has joined #openstack-infra | 19:33 | |
| *** Guest78215 has quit IRC | 19:34 | |
| *** salv-orlando has quit IRC | 19:39 | |
| *** apoorvad has quit IRC | 19:40 | |
| *** salv-orlando has joined #openstack-infra | 19:42 | |
| *** cody-somerville has quit IRC | 19:46 | |
| mtreinish | clarkb: you're not in -qa, do you have any ideas on how to make: https://review.openstack.org/#/c/311919/ faster | 19:47 |
| mtreinish | right now it's taking about 4min running from my laptop | 19:47 |
| *** hdaniel has joined #openstack-infra | 19:47 | |
| mtreinish | I'm thinking we might not be able to do that dynamically | 19:47 |
| *** inc0 has quit IRC | 19:48 | |
| *** piet has joined #openstack-infra | 19:49 | |
| *** teran has quit IRC | 19:52 | |
| *** igorbelikov has quit IRC | 19:52 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: builders: update cmake to match CMake plugin versions 2.x https://review.openstack.org/279754 | 19:54 |
| *** igorbelikov has joined #openstack-infra | 19:54 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add reno jobs for puppet-ceph https://review.openstack.org/313677 | 19:54 |
| *** teran has joined #openstack-infra | 19:55 | |
| *** thorst_ has quit IRC | 19:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Add ubuntu-precise to bindep-fallback https://review.openstack.org/313626 | 19:56 |
| *** thorst_ has joined #openstack-infra | 19:57 | |
| *** dprince has quit IRC | 19:59 | |
| *** thorst_ has quit IRC | 20:00 | |
| amrith | mordred, don't think it is possible to do what I suggested without access to the shell | 20:00 |
| amrith | and I don't think rax gives you that. | 20:00 |
| openstackgerrit | Merged openstack-infra/project-config: Replace bare-precise with ubuntu-precise in infra-puppet-apply-jobs https://review.openstack.org/313627 | 20:01 |
| *** jheroux has quit IRC | 20:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise https://review.openstack.org/313628 | 20:01 |
| pabelanger | AJaeger: thanks! | 20:02 |
| *** thorst_ has joined #openstack-infra | 20:02 | |
| mordred | amrith: they do not | 20:03 |
| mordred | amrith: I have spun up an ha group at rax | 20:03 |
| mordred | although I have not played with it much yet | 20:03 |
| mordred | but I'll report back when I do | 20:04 |
| amrith | thx mordred | 20:04 |
| amrith | am curious how to get around that now. | 20:04 |
| pabelanger | mordred: do you have a moment to check the FIPs in osic? I think they are leaking | 20:05 |
| *** dmk0202 has joined #openstack-infra | 20:05 | |
| mordred | pabelanger: yup! | 20:05 |
| mordred | why yes! there are many leaked fips | 20:06 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add semi-colon as separator for parsing package names https://review.openstack.org/313686 | 20:06 |
| *** kjorgensen has quit IRC | 20:06 | |
| pabelanger | Ya, osic has an issue this morning. I tried cleaning them up once before, but looks like they leaked out again | 20:06 |
| mordred | pabelanger: Cleaned 58 FIPs from osic-cloud1 | 20:07 |
| mordred | Cleaned 22 FIPs from bluebox-sjc1 | 20:07 |
| openstackgerrit | Stephen Lowrie proposed openstack/os-testr: Add subunit-describe-calls https://review.openstack.org/312275 | 20:07 |
| pabelanger | mordred: okay, just rechecking 308376 your FIP script | 20:07 |
| pabelanger | will try that next time | 20:08 |
| mordred | pabelanger: http://paste.openstack.org/show/496382 and http://paste.openstack.org/show/496383 | 20:08 |
| mordred | pabelanger: woot | 20:08 |
| *** javeriak has quit IRC | 20:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Set min-ready 0 for bare-precise https://review.openstack.org/313629 | 20:10 |
| *** thorst_ has quit IRC | 20:11 | |
| *** salv-orlando has quit IRC | 20:13 | |
| *** thorst_ has joined #openstack-infra | 20:13 | |
| *** twm2016 has joined #openstack-infra | 20:13 | |
| *** apoorvad has joined #openstack-infra | 20:13 | |
| *** aysyd has quit IRC | 20:14 | |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: docs: add missing :jenkins-wiki: macro to CMake builder https://review.openstack.org/313688 | 20:15 |
| twm2016 | This is kind of a reach here but I'm working a unit test refactoring from mox to mock. Does anyone know of any guides for this? I don't know too much about either of the projects but I want to learn. I found some side by side comparisons for basic commands in mox and mock but that doesn't help much when I get into the code. | 20:16 |
| openstackgerrit | Merged openstack-infra/project-config: Add VirtualBMC to gerritbot https://review.openstack.org/313280 | 20:16 |
| openstackgerrit | Stephen Lowrie proposed openstack/os-testr: Add subunit-describe-calls https://review.openstack.org/312275 | 20:17 |
| AJaeger | twm2016: this is the infra channel, not a generic development one. Better ask on #openstack-dev | 20:17 |
| *** thorst_ has quit IRC | 20:18 | |
| *** twm2016 has left #openstack-infra | 20:18 | |
| *** thorst_ has joined #openstack-infra | 20:19 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Ironic functional tests in Identity v3-only https://review.openstack.org/313690 | 20:19 |
| *** cody-somerville has joined #openstack-infra | 20:20 | |
| *** keedya has joined #openstack-infra | 20:24 | |
| *** thorst_ has quit IRC | 20:24 | |
| *** hdaniel has quit IRC | 20:24 | |
| *** thorst_ has joined #openstack-infra | 20:25 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Trove client didn't have release notes in mitaka https://review.openstack.org/313692 | 20:26 |
| *** thorst_ has quit IRC | 20:26 | |
| *** thorst_ has joined #openstack-infra | 20:26 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Trove client didn't have release notes in mitaka https://review.openstack.org/313692 | 20:28 |
| *** pt_15 has joined #openstack-infra | 20:29 | |
| *** Sukhdev has joined #openstack-infra | 20:29 | |
| *** e0ne has joined #openstack-infra | 20:29 | |
| pabelanger | #success bare-precise has been replaced by ubuntu-precise. Long live DIB | 20:30 |
| openstackstatus | pabelanger: Added success to Success page | 20:30 |
| *** kushal has quit IRC | 20:30 | |
| *** BigWillie has quit IRC | 20:30 | |
| *** openstackgerrit has quit IRC | 20:32 | |
| *** openstackgerrit has joined #openstack-infra | 20:33 | |
| *** ddieterly is now known as ddieterly[away] | 20:34 | |
| *** rfolco has quit IRC | 20:35 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: docs: add missing :jenkins-wiki: macro to CMake builder https://review.openstack.org/313688 | 20:35 |
| * dhellmann makes a note to nominate pabelanger for Most Creative Notice Author contributor award this cycle | 20:37 | |
| morgan | dhellmann: ++ | 20:37 |
| morgan | ;) | 20:37 |
| dhellmann | morgan : he's on a roll today | 20:37 |
| morgan | dhellmann: so, i am officially all setup with red hat now :) | 20:38 |
| morgan | dhellmann: and getting ready to hop a plane back home ^_^ | 20:38 |
| *** cody-somerville has quit IRC | 20:38 | |
| dhellmann | morgan : did you get a fedora or are they still out? | 20:38 |
| morgan | dhellmann: still out | 20:38 |
| dhellmann | bummer | 20:38 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-projects option https://review.openstack.org/293472 | 20:38 |
| dhellmann | morgan : well, welcome to the team! | 20:38 |
| pabelanger | dhellmann: Ha, thanks | 20:38 |
| morgan | dhellmann: sounded like quite the trial to get those hats. | 20:38 |
| *** e0ne has quit IRC | 20:39 | |
| pabelanger | morgan: welcome! | 20:39 |
| *** flwang1 has quit IRC | 20:39 | |
| *** jsavak has quit IRC | 20:39 | |
| *** e0ne has joined #openstack-infra | 20:40 | |
| *** e0ne has quit IRC | 20:40 | |
| *** Sukhdev has quit IRC | 20:40 | |
| *** jsavak has joined #openstack-infra | 20:40 | |
| *** Sukhdev has joined #openstack-infra | 20:41 | |
| morgan | pabelanger: ^_^ | 20:41 |
| smcginnis_ | So maybe you smart people can suggest what to do here. | 20:41 |
| *** kgiusti has quit IRC | 20:41 | |
| smcginnis_ | We have a non-voting python-cinderclient job failing for pypy. | 20:41 |
| smcginnis_ | Turns out the paramiko library updates recently to require cryptography 1.1. | 20:41 |
| smcginnis_ | Which has an explicit check to not allow pypy < 2.6. | 20:42 |
| smcginnis_ | Our pypy test I believe just installs the pypy version for the given platform. | 20:42 |
| openstackgerrit | Merged openstack-infra/project-config: add reno jobs for puppet-ceph https://review.openstack.org/313677 | 20:42 |
| smcginnis_ | Which for trusty is 2.2.1. | 20:42 |
| smcginnis_ | So I'm wondering - is there either a way to require a later pypy? Or is there a better solution to this? | 20:43 |
| fungi | i had one of those red fedoras years ago, but eventually gave it away. was promotional swag because my employer bought so many support licenses | 20:43 |
| fungi | like, somewhere back around 1999/2000 | 20:44 |
| *** gordc has quit IRC | 20:44 | |
| *** aysyd has joined #openstack-infra | 20:44 | |
| fungi | smcginnis_: could pin to paramiko<2 maybe? though rsn we'll be running newton jobs on ubuntu 16.04 lts which likely has new enough pypy | 20:45 |
| smcginnis_ | fungi, Yeah, 16.04 might be the answer. At least to get it passing in gate. | 20:46 |
| smcginnis_ | I'd hate to cap paramiko since there's good reasons to upgrade it for security updates. | 20:46 |
| fungi | yep, though also as you noted it drags in cryptography, which makes pip install without prebuilt wheels more painful (needs to link against local headers for openssl, cffi, et cetera) | 20:48 |
| smcginnis_ | I closed the tab now, but I believe Xenial includes the newer pypy by default, so I'm assuming it has all the necessary dependencies included in that. | 20:49 |
| *** baoli_ has quit IRC | 20:50 | |
| *** thorst_ has quit IRC | 20:52 | |
| *** thorst_ has joined #openstack-infra | 20:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Improve pot publishing https://review.openstack.org/313698 | 20:53 |
| *** grauzikas has quit IRC | 20:53 | |
| AJaeger | project-config cores, I missed that some repos have duplicated pot files in different paths. Could you review and merge the change above quickly, please? ^ | 20:54 |
| *** grauzikas has joined #openstack-infra | 20:54 | |
| fungi | AJaeger: yep, saw the ml thread. wondered what the fix might be for that | 20:55 |
| AJaeger | once the change is in, http://tarballs.openstack.org/translation-source/ should be deleted - we update it with each change, so brute force is fine. | 20:56 |
| AJaeger | fungi, please check my logic. Hope it's fine now. I gave a lengthy comment... | 20:57 |
| fungi | AJaeger: hemanthm is also looking at making a periodic job to generate glance sample configs, so i pointed him at your potfile work as an example of how we're getting away from pushing things like that into git repos and suggested maybe publishing them on tarballs.o.o instead like that | 20:58 |
| *** ddieterly[away] is now known as ddieterly | 20:58 | |
| *** jsavak has quit IRC | 20:58 | |
| *** dmk0202 has quit IRC | 20:58 | |
| AJaeger | fungi, the suggestion to use same path is the best IMHO, see https://translate.openstack.org/iteration/view/horizon/stable-mitaka/documents which shows the path | 20:58 |
| *** amitgandhinz has quit IRC | 20:58 | |
| AJaeger | fungi, yeah, this should be fine and we could reuse this for other repos that want sample configs | 20:59 |
| *** ddieterly is now known as ddieterly[away] | 20:59 | |
| *** pcrews__ has joined #openstack-infra | 20:59 | |
| AJaeger | regarding proposal jobs: https://review.openstack.org/310929 is the project-config review guideline, it mentions those as we discussed in Austin. | 21:00 |
| pabelanger | Here is an interesting exception from nodepool: http://paste.openstack.org/show/496390/ | 21:01 |
| *** thorst_ has quit IRC | 21:01 | |
| AJaeger | anteaya: thanks for review. | 21:01 |
| anteaya | AJaeger: you are welcome | 21:01 |
| *** Sukhdev has quit IRC | 21:02 | |
| *** pcrews_ has quit IRC | 21:02 | |
| *** ddieterly[away] is now known as ddieterly | 21:02 | |
| *** thorst_ has joined #openstack-infra | 21:03 | |
| fungi | AJaeger: the potfile fix for horizon looks fine to me | 21:05 |
| fungi | AJaeger: and you need all of translation-source deleted from tarballs.o.o? | 21:06 |
| *** baoli has joined #openstack-infra | 21:06 | |
| AJaeger | fungi, yes, remove the whole directory... | 21:06 |
| AJaeger | We'll populate it again ;) | 21:07 |
| AJaeger | thanks, fungi! | 21:07 |
| fungi | done (well, moved to my homedir temporarily in case you need it back for some reason) | 21:07 |
| *** kzaitsev_mb has quit IRC | 21:07 | |
| fungi | though i guess we might need to do that again if anything merges to trigger another update between now and when that change propagates to the worker | 21:07 |
| AJaeger | I shouldn't need it - it's all created data... But good to know - feel free to delete on Monday if I didn't ask for it. | 21:08 |
| *** thorst_ has quit IRC | 21:08 | |
| AJaeger | yeah, this might need redoing, let's see | 21:08 |
| *** ddieterly is now known as ddieterly[away] | 21:08 | |
| fungi | also, rackspace support just triaged our gerrit db stuck resize ticket, escalating it to their trove group looks like | 21:08 |
| fungi | no details yet | 21:08 |
| fungi | i'm going to pop out for an early dinner, then check back in and start the pvmove of /home/gerrit2 once i get back | 21:09 |
| AJaeger | thanks, fungi! | 21:09 |
| dims | pabelanger fungi : quick question, is bandersnatch back on its feet yet? | 21:09 |
| dims | and have a good weekend :) | 21:09 |
| *** thorst has joined #openstack-infra | 21:10 | |
| pabelanger | dims: yes, we've switched to the RW mirror for the time being | 21:10 |
| dims | pabelanger : so the sync should be complete by now? | 21:10 |
| pabelanger | dims: They won't be atomic if I understand correctly, but they are up to date | 21:10 |
| dims | pabelanger : ack i'll kick the tires | 21:11 |
| dims | thanks! | 21:11 |
| pabelanger | dims: sure, let us know if there are problems | 21:11 |
| dims | you bet pabelanger ! | 21:11 |
| *** ddieterly[away] is now known as ddieterly | 21:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Improve pot publishing https://review.openstack.org/313698 | 21:13 |
| *** psilvad has quit IRC | 21:13 | |
| *** MarkAtwood has quit IRC | 21:14 | |
| zigo | pabelanger: Hi there! | 21:14 |
| pabelanger | zigo: hello | 21:15 |
| zigo | pabelanger: It's been a week since AJaeger helped me with this: https://review.openstack.org/#/c/311190/ | 21:15 |
| zigo | (on the Friday at the summit) | 21:15 |
| zigo | So probably now you can have a look? | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| zigo | Nobody reviewed it yet... | 21:15 |
| pabelanger | zigo: sure, I'll check it out this weekend | 21:16 |
| *** baoli has quit IRC | 21:16 | |
| zigo | Cheers. | 21:16 |
| zigo | mordred: Maybe you could look into it too? | 21:16 |
| *** xyang1 has quit IRC | 21:16 | |
| zigo | pabelanger: How will the other side (ie: the pick-up job) should be done? | 21:16 |
| pabelanger | zigo: I can write the script to do that. Looks like you are using the git sha1 hash, so should be straight forward | 21:17 |
| zigo | I did yes. | 21:17 |
| pabelanger | zigo: we likely want to move the upload par out of your script, since other jobs could use it. But should be safe place to start | 21:17 |
| zigo | So, tarballs/debian/deb-openstack-pkg-tools/<sha256>/* | 21:17 |
| *** roxanaghe has joined #openstack-infra | 21:18 | |
| pabelanger | Ya, I just need to see what the directory script on tarballs.o.o will look like | 21:18 |
| pabelanger | we likely want tarballs.o.o/deb-openstack-pkg-toosl/<sha256>/* | 21:18 |
| pabelanger | minus the typo of tools | 21:19 |
| zigo | pabelanger: Something like this: http://mitaka-jessie.pkgs.mirantis.com/debian/pool/jessie-mitaka-backports/main/o/openstack-pkg-tools/ | 21:19 |
| zigo | :) | 21:19 |
| pabelanger | yup | 21:19 |
| *** thorst has quit IRC | 21:19 | |
| zigo | Minus the .build file, which I'm deleting on purpose (we already have the build log, it'd be redondant...) | 21:19 |
| *** ayoung has quit IRC | 21:20 | |
| *** roxanaghe has quit IRC | 21:20 | |
| *** MarkAtwood has joined #openstack-infra | 21:20 | |
| *** thorst has joined #openstack-infra | 21:20 | |
| zigo | pabelanger: What was the thing you were working on this week btw? | 21:20 |
| *** regXboi has quit IRC | 21:21 | |
| zigo | For me, I've been sick all the way since I'm back from the summit (right after I reached home), so I didn't do anything... :( | 21:21 |
| *** roxanaghe has joined #openstack-infra | 21:22 | |
| *** Sukhdev has joined #openstack-infra | 21:23 | |
| *** eharney has quit IRC | 21:24 | |
| *** aysyd has quit IRC | 21:24 | |
| *** lakshmiS has quit IRC | 21:24 | |
| *** ddieterly has quit IRC | 21:28 | |
| *** thorst has quit IRC | 21:28 | |
| *** vilobhmm11 has joined #openstack-infra | 21:28 | |
| *** ewilson has quit IRC | 21:29 | |
| *** vilobhmm11 has quit IRC | 21:30 | |
| *** vilobhmm11 has joined #openstack-infra | 21:30 | |
| *** matt-borland has quit IRC | 21:32 | |
| *** vilobhmm11 has quit IRC | 21:33 | |
| *** leakypipes has quit IRC | 21:35 | |
| *** jcoufal has quit IRC | 21:35 | |
| *** thorst has joined #openstack-infra | 21:36 | |
| EmilienM | AJaeger: you never sleep! | 21:37 |
| EmilienM | thanks for the quick reviews | 21:37 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:38 | |
| *** mminesh has joined #openstack-infra | 21:39 | |
| *** thorst has quit IRC | 21:40 | |
| mminesh | Trying to run git review for the first time and getting "fatal: ICLA contributor agreement requires current contact information. | 21:40 |
| mminesh | Please review your contact information: | 21:40 |
| mminesh | https://review.openstack.org/#/settings/contact" | 21:40 |
| mminesh | I have tried updating the info on the link but getting server error.... | 21:40 |
| mminesh | any help? | 21:40 |
| mminesh | test was run against the sandbox project before i perform any actual commit..but i have been unsuccessful so far | 21:41 |
| *** thorst has joined #openstack-infra | 21:41 | |
| anteaya | mminesh: did you follow all the steps here: http://docs.openstack.org/infra/manual/developers.html#account-setup | 21:42 |
| *** thorst has quit IRC | 21:43 | |
| anteaya | mminesh: if so then read this https://ask.openstack.org/en/question/3413/icla-contributor-agreement-requires-current-contact-information-error-while-using-git-review/ | 21:43 |
| *** thorst has joined #openstack-infra | 21:43 | |
| *** mriedem has quit IRC | 21:44 | |
| *** vgridnev has joined #openstack-infra | 21:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Really fix translations copy https://review.openstack.org/313707 | 21:44 |
| AJaeger | EmilienM: I wanted to - but first needed to fix translation copying ;) | 21:45 |
| AJaeger | fungi, anteaya see 313707, hope that works better now... | 21:45 |
| AJaeger | Currently all translation pushes fail ;( | 21:45 |
| *** amrith is now known as _amrith_ | 21:45 | |
| *** thiagop has quit IRC | 21:47 | |
| AJaeger | project-config cores, please review 313707 for translation push - sorry, I missed the zanata-cli behaviour of looking into all directories including .directories ;( | 21:47 |
| * AJaeger calls it a day... | 21:48 | |
| AJaeger | thanks & bye | 21:48 |
| *** HeOS has joined #openstack-infra | 21:48 | |
| *** salv-orlando has joined #openstack-infra | 21:51 | |
| *** pradk has quit IRC | 21:52 | |
| *** davideagnello has quit IRC | 21:53 | |
| *** MarkAtwood has quit IRC | 21:57 | |
| *** tphummel has quit IRC | 21:58 | |
| *** piet has quit IRC | 21:59 | |
| openstackgerrit | Merged openstack/requirements: Require Babel>=2.3.4 https://review.openstack.org/311761 | 22:01 |
| *** thorst has quit IRC | 22:04 | |
| *** ewilson has joined #openstack-infra | 22:07 | |
| *** dynarro has quit IRC | 22:09 | |
| *** Sukhdev has quit IRC | 22:10 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs https://review.openstack.org/313717 | 22:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/313721 | 22:16 |
| *** ayoung has joined #openstack-infra | 22:16 | |
| *** ewilson has quit IRC | 22:17 | |
| *** thorst has joined #openstack-infra | 22:17 | |
| *** ewilson has joined #openstack-infra | 22:17 | |
| *** rhallisey has quit IRC | 22:18 | |
| *** vilobhmm11 has joined #openstack-infra | 22:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr: Updated from global requirements https://review.openstack.org/313749 | 22:21 |
| *** thorst has quit IRC | 22:22 | |
| *** dynarro has joined #openstack-infra | 22:22 | |
| *** keedya has quit IRC | 22:24 | |
| *** tonytan4ever has quit IRC | 22:24 | |
| *** tphummel has joined #openstack-infra | 22:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:32 | |
| *** roxanaghe has quit IRC | 22:33 | |
| *** vgridnev has quit IRC | 22:34 | |
| *** david-lyle has quit IRC | 22:35 | |
| *** ewilson has quit IRC | 22:37 | |
| kevinbenton | is someone around familiar with devstack? i'm troubleshooting some heat gate failures and i have the issue tracked down to the recent removal of the VPNaaS code from devstack | 22:37 |
| *** david-lyle has joined #openstack-infra | 22:37 | |
| *** vhosakot has quit IRC | 22:45 | |
| *** sdake has joined #openstack-infra | 22:45 | |
| *** markvoelker has quit IRC | 22:47 | |
| *** sdake_ has joined #openstack-infra | 22:48 | |
| *** dynarro has quit IRC | 22:48 | |
| *** markvoelker has joined #openstack-infra | 22:48 | |
| *** rbrndt has quit IRC | 22:49 | |
| *** sdake has quit IRC | 22:50 | |
| *** ewilson has joined #openstack-infra | 22:51 | |
| samueldmq | anteaya: sorry I was afk earlier, thanks for the heads up | 22:52 |
| *** r-daneel has quit IRC | 22:52 | |
| samueldmq | anteaya: re: patch 264398 | 22:52 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs https://review.openstack.org/313717 | 22:53 |
| fungi | kevinbenton: you may have more luck asking in #openstack-qa since devstack is a qa project | 22:54 |
| kevinbenton | fungi: thx | 22:55 |
| *** sdake has joined #openstack-infra | 22:55 | |
| anteaya | samueldmq: sure | 22:55 |
| *** jamesmca_ has joined #openstack-infra | 22:56 | |
| *** sdake_ has quit IRC | 22:56 | |
| fungi | so... gerrit database resize update is that apparently the resize succeeded but for some reason status metadata didn't update correctly so the api is still showing it as resizing | 22:57 |
| fungi | i'm going ahead with the pvmove for our gerrit data now | 22:57 |
| *** tpsilva has quit IRC | 22:57 | |
| fungi | reminder that it's running in a root screen session on review.o.o | 22:57 |
| fungi | already 1% done so this may not take long | 22:58 |
| *** signed8bit is now known as signed8bit_Zzz | 22:58 | |
| *** _amrith_ is now known as amrith | 23:00 | |
| fungi | 10% done | 23:00 |
| *** jamesmca_ has quit IRC | 23:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Really fix translations copy https://review.openstack.org/313707 | 23:02 |
| JayF | If someone has a spare moment, https://review.openstack.org/#/c/313637 is a one line change to add pyghmi to projects for Ironic jobs so a new job we added to test it will pass | 23:03 |
| JayF | already has 1x+2 just needs one more. | 23:03 |
| *** ams_ has quit IRC | 23:03 | |
| *** yamahata has quit IRC | 23:04 | |
| *** sdake has quit IRC | 23:05 | |
| *** ewilson has quit IRC | 23:06 | |
| *** sridhar_ram has quit IRC | 23:08 | |
| *** roxanaghe has joined #openstack-infra | 23:09 | |
| *** mixos has quit IRC | 23:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Add openstack/pyghmi to the $PROJECTS variable https://review.openstack.org/313637 | 23:11 |
| *** daemontool has quit IRC | 23:12 | |
| *** signed8bit_Zzz is now known as signed8bit | 23:13 | |
| fungi | JayF: ^ | 23:13 |
| JayF | thanks a bunch fungi, I know lucas will be glad that's merged when he gets back | 23:14 |
| fungi | lucas-afk: ^ | 23:15 |
| *** ddieterly has joined #openstack-infra | 23:16 | |
| fungi | we're already at 70% on the pvmove, so should be done soon and then i can clean up the old pv and expand the lv | 23:17 |
| openstackgerrit | David Lyle proposed openstack/requirements: Remove XStatic-QUnit https://review.openstack.org/313795 | 23:18 |
| *** e has joined #openstack-infra | 23:22 | |
| *** ddieterly is now known as ddieterly[away] | 23:24 | |
| *** Jeffrey4l has joined #openstack-infra | 23:26 | |
| *** tphummel has quit IRC | 23:27 | |
| *** kzaitsev_mb has quit IRC | 23:29 | |
| openstackgerrit | David Lyle proposed openstack/requirements: Remove XStatic-Angular-FileUpload https://review.openstack.org/313797 | 23:30 |
| *** sdake has joined #openstack-infra | 23:34 | |
| *** edmondsw has quit IRC | 23:34 | |
| *** xarses has quit IRC | 23:34 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu binary gate on liberty branch https://review.openstack.org/313561 | 23:34 |
| *** akshai has quit IRC | 23:35 | |
| *** signed8bit is now known as signed8bit_Zzz | 23:36 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Explicitly define if ipxe is enabled or not https://review.openstack.org/313799 | 23:37 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs https://review.openstack.org/313800 | 23:37 |
| *** sdake_ has joined #openstack-infra | 23:37 | |
| *** sdake has quit IRC | 23:38 | |
| *** rossella_s has quit IRC | 23:40 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Make oraclelinux jobs for Kolla gate always run https://review.openstack.org/311803 | 23:41 |
| *** rossella_s has joined #openstack-infra | 23:41 | |
| pabelanger | clarkb: I figure out the issue with nodepool. We incorrectly set the image-update, we inverted the minutes and hours | 23:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Run image-update at 10:34hrs UTC https://review.openstack.org/313802 | 23:45 |
| pabelanger | clarkb: ^ | 23:45 |
| anteaya | fungi: ack | 23:46 |
| openstackgerrit | David Lyle proposed openstack/requirements: Remove XStatic-JQuery.Bootstrap.Wizard https://review.openstack.org/313803 | 23:46 |
| JayF | What is the difference/split between zuul/layout.yaml and jenkins/jobs/projects.yaml | 23:46 |
| JayF | It seems like they contain somewhat duplicated information? | 23:46 |
| fungi | and it's done, so removing the old volume from the vg and then detaching it from the instance | 23:46 |
| anteaya | pabelanger: ah it wasn't getting to the 34th hour, hence was not updating | 23:47 |
| pabelanger | JayF: one is zuul configuration, the other is jenkins. We use 2 different applications to process the configuration | 23:47 |
| anteaya | fungi: yay done | 23:47 |
| fungi | JayF: one configures uul (our job scheduler) and the other configures jenkins (our job runner). different pieces of software | 23:47 |
| fungi | s/uul/zuul/ | 23:47 |
| JayF | But the configs should be relatively in sync then, right? | 23:47 |
| pabelanger | anteaya: Ya, its running now. So, 34hrs since last update | 23:47 |
| * anteaya hands fungi a z to put at the front of the uul | 23:47 | |
| anteaya | pabelanger: oh, okay now I understand | 23:48 |
| fungi | JayF: jenkins job configurations are the details of what runs, zuul configuration says when to run them | 23:48 |
| JayF | this makes sense with why my change was failing, I think I can fix this up, cool | 23:48 |
| pabelanger | anteaya: ya, confusing for sure. I just noticed it running again | 23:48 |
| *** woodster_ has quit IRC | 23:48 | |
| anteaya | pabelanger: good eye | 23:48 |
| fungi | JayF: with zuul v3 (and to some extend 2.5) this gets less confusing as jenkins will be out of the picture | 23:49 |
| fungi | s/extend/extent/ | 23:49 |
| fungi | i can't type tonight | 23:49 |
| JayF | yeah, I think I'm more confused now :/ | 23:49 |
| JayF | the "project" something is defined under in jenkins/jobs/projects.yaml don't matter, it seems? | 23:49 |
| *** cody-somerville has joined #openstack-infra | 23:49 | |
| fungi | JayF: right, it's a logical aggregation within jenkns | 23:49 |
| fungi | and somewhat arbitrary | 23:50 |
| fungi | we have "projects" defining jobs which run under other git repos than their own in many cases | 23:50 |
| JayF | So my patch I thought was done, I'm just getting to the hard part it seems | 23:51 |
| JayF | so going to save that for later, I think | 23:51 |
| *** gyee has quit IRC | 23:53 | |
| *** Guest34798 has quit IRC | 23:54 | |
| *** |-paul-| has joined #openstack-infra | 23:55 | |
| *** signed8bit_Zzz is now known as signed8bit | 23:55 | |
| fungi | okay, i've got /home/gerrit2 moved onto a 200gb ssd cinder volume and expanded so that it's now only ~47% full. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2839&rra_id=all should reflect that shortly | 23:55 |
| *** salv-orlando has quit IRC | 23:56 | |
| fungi | i've also cleaned up and detached/deleted the old 100gb volume | 23:56 |
| anteaya | thank you | 23:56 |
| *** akshai has joined #openstack-infra | 23:57 | |
| pabelanger | fungi: will have to check history on review.o.o to see how you did that with out stopping gerrit | 23:57 |
| anteaya | cacti indeed reflects the change | 23:57 |
| fungi | #status log the Review-MySQL trove instance has now been expanded to 50gb (19% full) and /home/gerrit2 on review.openstack.org increased to 200gb (47% full) | 23:57 |
| openstackstatus | fungi: finished logging | 23:57 |
| fungi | pabelanger: create new cinder volume and have nova attach it, check dmesg to see what the device name was since you can't trust the api | 23:57 |
| openstackgerrit | Kevin Benton proposed openstack-infra/project-config: Use neutron-vpnaas grenade plugin in grenade jobs https://review.openstack.org/313806 | 23:58 |
| fungi | old 100gb volume was /dev/xvdb and new one was /dev/xvdc | 23:58 |
| pabelanger | fungi: Ah, that makes sense | 23:58 |
| pabelanger | clouds! | 23:58 |
| fungi | commands i ran on review.o.o were as follows: | 23:58 |
| *** akshai_ has joined #openstack-infra | 23:58 | |
| fungi | sudo parted /dev/xvdc1 mklabel msdos mkpart primary 0% 100% set 1 lvm on | 23:59 |
| fungi | sudo pvcreate /dev/xvdc1 | 23:59 |
| fungi | sudo vgextend main /dev/xvdc1 | 23:59 |
| fungi | sudo pvmove /dev/xvdb1 /dev/xvdc1 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
