| *** dina_belova has joined #openstack-infra | 00:11 | |
| *** dina_belova has quit IRC | 00:15 | |
| fungi | pew pew scrollback lasers | 00:20 |
|---|---|---|
| *** dkliban has quit IRC | 00:38 | |
| *** jesusaurus has quit IRC | 00:39 | |
| *** jesusaurus has joined #openstack-infra | 00:44 | |
| *** colinmcnamara1 has joined #openstack-infra | 00:48 | |
| *** colinmcnamara has quit IRC | 00:48 | |
| *** erfanian has joined #openstack-infra | 00:50 | |
| *** dina_belova has joined #openstack-infra | 01:11 | |
| *** CaptTofu has joined #openstack-infra | 01:14 | |
| *** dina_belova has quit IRC | 01:16 | |
| *** tianst20 has quit IRC | 01:29 | |
| *** xchu has joined #openstack-infra | 01:48 | |
| *** DennyZhang has joined #openstack-infra | 01:56 | |
| *** yaguang has joined #openstack-infra | 01:57 | |
| *** yaguang has quit IRC | 02:05 | |
| clarkb | mordred: fungi: jeblair: it just occurred to me that we could potentially cdn our git repos on git.o.o. to make life better for lifeless and other people with high latency | 02:09 |
| *** dina_belova has joined #openstack-infra | 02:12 | |
| lifeless | clarkb: meh. git is very good for keeping a cache; we've got comprehensive support for caching it in dib | 02:13 |
| clarkb | ya git being distributed makes it less painful | 02:15 |
| lifeless | also unless you had a git server cdn it would be worse | 02:16 |
| lifeless | in big projects when a big pack occurs | 02:16 |
| lifeless | e.g. nova | 02:16 |
| *** zul has quit IRC | 02:16 | |
| *** dina_belova has quit IRC | 02:16 | |
| *** yaguang has joined #openstack-infra | 02:19 | |
| *** zul has joined #openstack-infra | 02:20 | |
| *** zul has quit IRC | 02:34 | |
| *** CaptTofu has quit IRC | 02:35 | |
| *** CaptTofu has joined #openstack-infra | 02:36 | |
| *** DennyZhang has quit IRC | 02:38 | |
| *** zul has joined #openstack-infra | 02:42 | |
| *** erfanian has quit IRC | 02:51 | |
| *** sgviking has joined #openstack-infra | 02:58 | |
| *** mriedem has quit IRC | 03:00 | |
| *** tian has joined #openstack-infra | 03:02 | |
| *** yaguang has quit IRC | 03:03 | |
| tian | hi, my evn : ubuntu 12.04 . when i use cmd "mvn clean generate-sources" , there is a error "Plugin org.apache.maven.plugins:maven-clean-plugin:2.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.3: Could not transfer artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.3 from/to rackspace-research (http://maven.research.rackspacecloud.com/conte | 03:11 |
| tian | nt/groups/public/): Connection to http://maven.research.rackspacecloud.com refused: Connection timed out -> [Help 1] | 03:11 |
| tian | " | 03:11 |
| *** dina_belova has joined #openstack-infra | 03:12 | |
| tian | anybody know why ? | 03:12 |
| clarkb | tian connection timed out to that rackspace url. looks like network related | 03:14 |
| *** yaguang has joined #openstack-infra | 03:16 | |
| *** colinmcnamara1 has quit IRC | 03:16 | |
| *** dina_belova has quit IRC | 03:17 | |
| *** yaguang has quit IRC | 03:27 | |
| *** yaguang has joined #openstack-infra | 03:36 | |
| *** dkliban has joined #openstack-infra | 03:40 | |
| *** rcleere has joined #openstack-infra | 03:49 | |
| *** colinmcnamara has joined #openstack-infra | 03:53 | |
| *** colinmcnamara has quit IRC | 03:55 | |
| *** vogxn has joined #openstack-infra | 03:56 | |
| *** Ryan_Lane has joined #openstack-infra | 03:58 | |
| *** Ryan_Lane has quit IRC | 04:03 | |
| *** Ryan_Lane has joined #openstack-infra | 04:04 | |
| *** Ryan_Lane has quit IRC | 04:04 | |
| *** Ryan_Lane has joined #openstack-infra | 04:04 | |
| *** dina_belova has joined #openstack-infra | 04:13 | |
| *** dina_belova has quit IRC | 04:17 | |
| *** UtahDave has joined #openstack-infra | 04:19 | |
| *** Alex_Gaynor has quit IRC | 04:19 | |
| *** mkoderer has quit IRC | 04:19 | |
| *** Alex_Gaynor has joined #openstack-infra | 04:23 | |
| *** dkliban has quit IRC | 04:28 | |
| *** dkliban has joined #openstack-infra | 04:34 | |
| *** nati_ueno has joined #openstack-infra | 04:36 | |
| *** rcleere has quit IRC | 04:40 | |
| *** mberwanger has joined #openstack-infra | 04:41 | |
| *** boris-42 has joined #openstack-infra | 04:53 | |
| *** dkliban has quit IRC | 05:08 | |
| *** mberwanger has quit IRC | 05:10 | |
| *** dina_belova has joined #openstack-infra | 05:13 | |
| *** nati_ueno has quit IRC | 05:15 | |
| *** thomasbiege1 has joined #openstack-infra | 05:15 | |
| *** vogxn has quit IRC | 05:17 | |
| *** thomasbiege1 has quit IRC | 05:17 | |
| *** dina_belova has quit IRC | 05:18 | |
| *** thomasbiege has joined #openstack-infra | 05:21 | |
| *** nati_ueno has joined #openstack-infra | 05:23 | |
| *** nicedice_ has quit IRC | 05:23 | |
| *** nati_ueno has quit IRC | 05:23 | |
| *** thomasbiege has quit IRC | 05:24 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:41 | |
| *** vogxn has joined #openstack-infra | 05:44 | |
| *** dina_belova has joined #openstack-infra | 05:54 | |
| *** ftcjeff has quit IRC | 06:03 | |
| *** ftcjeff has joined #openstack-infra | 06:04 | |
| *** vogxn1 has joined #openstack-infra | 06:05 | |
| *** thomasbiege1 has joined #openstack-infra | 06:06 | |
| *** vogxn has quit IRC | 06:06 | |
| *** nayward has quit IRC | 06:06 | |
| *** thomasbiege1 has quit IRC | 06:08 | |
| *** ftcjeff has quit IRC | 06:08 | |
| *** mkoderer has joined #openstack-infra | 06:21 | |
| *** jjmb has quit IRC | 06:22 | |
| *** dina_belova has quit IRC | 06:24 | |
| *** SergeyLukjanov has quit IRC | 06:25 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:30 | |
| *** kspear has quit IRC | 06:30 | |
| *** kspear has joined #openstack-infra | 06:32 | |
| *** SergeyLukjanov has quit IRC | 06:38 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:42 | |
| *** dina_belova has joined #openstack-infra | 06:43 | |
| *** SergeyLukjanov has quit IRC | 06:48 | |
| *** dina_belova has quit IRC | 06:57 | |
| *** DennyZhang has joined #openstack-infra | 06:58 | |
| *** DennyZhang has quit IRC | 07:12 | |
| *** vogxn has joined #openstack-infra | 07:15 | |
| *** fbo_away is now known as fbo | 07:16 | |
| *** odyssey4me3 has joined #openstack-infra | 07:16 | |
| chmouel | jeblair: what i about i put that in stackforge? or is that too small of a project for going there? | 07:19 |
| *** xchu has quit IRC | 07:22 | |
| *** jjmb has joined #openstack-infra | 07:23 | |
| *** nsaje1 has joined #openstack-infra | 07:23 | |
| *** xBsd has joined #openstack-infra | 07:24 | |
| *** Ryan_Lane has quit IRC | 07:26 | |
| *** xchu has joined #openstack-infra | 07:39 | |
| *** vogxn has quit IRC | 07:39 | |
| *** thomasbiege1 has joined #openstack-infra | 07:42 | |
| *** thomasbiege1 has quit IRC | 07:43 | |
| *** UtahDave has quit IRC | 07:52 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** Ryan_Lane has joined #openstack-infra | 07:57 | |
| *** jhesketh has quit IRC | 08:03 | |
| *** Ryan_Lane has quit IRC | 08:04 | |
| *** derekh has joined #openstack-infra | 08:05 | |
| *** dina_belova has joined #openstack-infra | 08:08 | |
| *** nayward has joined #openstack-infra | 08:11 | |
| *** dina_belova has quit IRC | 08:13 | |
| *** mkerrin has joined #openstack-infra | 08:13 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Add more deterministic port selection for Gerrit. https://review.openstack.org/42581 | 08:17 |
| *** boris-42 has quit IRC | 08:19 | |
| *** psedlak has joined #openstack-infra | 08:21 | |
| *** shardy_afk is now known as shardy | 08:39 | |
| *** tian has quit IRC | 08:48 | |
| *** dina_belova has joined #openstack-infra | 09:09 | |
| *** dina_belova has quit IRC | 09:13 | |
| *** ruhe has joined #openstack-infra | 09:14 | |
| *** morganfainberg is now known as morganfainberg|a | 09:27 | |
| *** moted has quit IRC | 09:30 | |
| *** boris-42 has joined #openstack-infra | 09:30 | |
| *** xBsd has quit IRC | 09:31 | |
| *** xBsd has joined #openstack-infra | 09:34 | |
| *** xchu has quit IRC | 09:34 | |
| *** xchu has joined #openstack-infra | 09:47 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack/requirements: Allow use of oslo.messaging 1.2.0a9 https://review.openstack.org/42596 | 09:47 |
| *** chuckieb|2 has quit IRC | 09:53 | |
| *** Ryan_Lane has joined #openstack-infra | 10:02 | |
| *** xchu has quit IRC | 10:09 | |
| *** dina_belova has joined #openstack-infra | 10:09 | |
| *** Ryan_Lane has quit IRC | 10:09 | |
| *** nayward has quit IRC | 10:11 | |
| *** dina_belova has quit IRC | 10:13 | |
| *** weshay has joined #openstack-infra | 10:14 | |
| *** moted has joined #openstack-infra | 10:15 | |
| *** DennyZhang has joined #openstack-infra | 10:15 | |
| *** ruhe has quit IRC | 10:16 | |
| *** pblaho has joined #openstack-infra | 10:21 | |
| *** DennyZhang has quit IRC | 10:23 | |
| *** yaguang has quit IRC | 10:24 | |
| *** nayward has joined #openstack-infra | 10:30 | |
| *** tian has joined #openstack-infra | 10:30 | |
| *** fbo is now known as fbo_away | 10:34 | |
| openstackgerrit | A change was merged to openstack/requirements: Allow use of oslo.messaging 1.2.0a9 https://review.openstack.org/42596 | 10:40 |
| *** nayward has quit IRC | 10:42 | |
| *** ruhe has joined #openstack-infra | 10:43 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add groups-staging.openstack.org node https://review.openstack.org/42608 | 10:44 |
| *** mikal has quit IRC | 10:45 | |
| *** mikal has joined #openstack-infra | 10:47 | |
| *** chuckieb has joined #openstack-infra | 10:49 | |
| *** psedlak_ has joined #openstack-infra | 11:03 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/config: Make oslo.messaging gating on devstack/tempest https://review.openstack.org/42613 | 11:07 |
| *** psedlak has quit IRC | 11:07 | |
| *** xBsd has quit IRC | 11:08 | |
| *** xBsd has joined #openstack-infra | 11:09 | |
| *** vogxn has joined #openstack-infra | 11:16 | |
| *** olaph has quit IRC | 11:19 | |
| *** vogxn has quit IRC | 11:21 | |
| *** olaph has joined #openstack-infra | 11:26 | |
| *** dmakogon__ has joined #openstack-infra | 11:27 | |
| *** Ryan_Lane has joined #openstack-infra | 11:27 | |
| *** GheRivero_CH is now known as GheRivero | 11:27 | |
| *** xBsd has quit IRC | 11:28 | |
| *** dmakogon_ has quit IRC | 11:31 | |
| *** Ryan_Lane has quit IRC | 11:31 | |
| *** fbo_away is now known as fbo | 11:37 | |
| *** thomasbiege1 has joined #openstack-infra | 11:40 | |
| *** weshay has quit IRC | 11:44 | |
| *** cody-somerville has joined #openstack-infra | 11:44 | |
| *** thomasbiege1 has quit IRC | 11:44 | |
| *** thomasbiege1 has joined #openstack-infra | 11:45 | |
| *** weshay has joined #openstack-infra | 11:50 | |
| *** dkliban has joined #openstack-infra | 11:52 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add groups-staging.openstack.org node https://review.openstack.org/42608 | 11:54 |
| *** dina_belova has joined #openstack-infra | 11:57 | |
| *** nsaje1 has quit IRC | 11:58 | |
| *** dina_belova has quit IRC | 11:58 | |
| *** rfolco has joined #openstack-infra | 11:58 | |
| *** weshay has quit IRC | 11:59 | |
| *** nayward has joined #openstack-infra | 11:59 | |
| *** lcestari has joined #openstack-infra | 12:01 | |
| *** apcruz has joined #openstack-infra | 12:04 | |
| *** CaptTofu has quit IRC | 12:05 | |
| *** pcm_ has joined #openstack-infra | 12:05 | |
| *** CaptTofu has joined #openstack-infra | 12:05 | |
| *** jjmb1 has joined #openstack-infra | 12:07 | |
| *** dina_belova has joined #openstack-infra | 12:07 | |
| *** psedlak__ has joined #openstack-infra | 12:10 | |
| *** jjmb has quit IRC | 12:10 | |
| *** dina_belova has quit IRC | 12:11 | |
| *** xBsd has joined #openstack-infra | 12:11 | |
| *** psedlak_ has quit IRC | 12:14 | |
| *** sandywalsh has quit IRC | 12:17 | |
| *** woodspa has joined #openstack-infra | 12:21 | |
| *** beagles has joined #openstack-infra | 12:24 | |
| *** dina_belova has joined #openstack-infra | 12:26 | |
| *** sandywalsh has joined #openstack-infra | 12:30 | |
| *** simonmcc has quit IRC | 12:31 | |
| *** simonmcc has joined #openstack-infra | 12:31 | |
| *** simonmcc has quit IRC | 12:31 | |
| *** nayward has quit IRC | 12:31 | |
| *** simonmcc has joined #openstack-infra | 12:31 | |
| *** simonmcc has quit IRC | 12:32 | |
| *** simonmcc has joined #openstack-infra | 12:32 | |
| *** simonmcc has quit IRC | 12:32 | |
| *** dina_belova has quit IRC | 12:32 | |
| *** simonmcc has joined #openstack-infra | 12:33 | |
| *** nayward has joined #openstack-infra | 12:35 | |
| *** mriedem has joined #openstack-infra | 12:36 | |
| pblaho | hi guys... I have a question w/r/t voting: false settings in zuul/layout.yaml ... | 12:37 |
| pblaho | I have seen a bunch of jobs with this settings... mainly python33 kind of jobs.... | 12:38 |
| pblaho | do these jobs even run or they just do not count on gerrit voting and/or when deciding if it will be merged... | 12:38 |
| *** thomasbiege1 has quit IRC | 12:42 | |
| mordred | chmouel: there are no projects that are too small :) | 12:44 |
| mordred | pblaho: they are run and their result does not count for gerrit voting/deciding | 12:45 |
| *** dprince has joined #openstack-infra | 12:46 | |
| chmouel | mordred: cool I will move it there, | 12:46 |
| chmouel | is there a wiki page handy that explains the process ? (moving a project into stackforge) | 12:48 |
| *** nayward has quit IRC | 12:50 | |
| *** nayward has joined #openstack-infra | 12:50 | |
| *** xqueralt has joined #openstack-infra | 12:50 | |
| pblaho | mordred: thank you.... so.. b/c I recently submited a change with python33 and pylint for stackforge/python-tuskarclient and now I discovered that it will not be that easy to be python3 compatible (oslo.config is not - for the package ver. 1.1.1 from pypi).... | 12:51 |
| *** thomasbiege1 has joined #openstack-infra | 12:51 | |
| pblaho | so the best solution is to set python33 job for tuskarclient to voting: false... ? | 12:51 |
| mordred | pblaho: yes. I think so | 12:51 |
| pblaho | mordred: ok, thanks... preparing change right now... | 12:51 |
| *** anteaya has joined #openstack-infra | 12:51 | |
| mordred | and then you can slowly work on getting python33 working, and then when it does, we can change it back to voting | 12:51 |
| *** thomasbiege1 has quit IRC | 12:52 | |
| pblaho | mrodden: yes... that is the way I want it... | 12:54 |
| pblaho | mrodden: sorry... | 12:54 |
| pblaho | mordred: ^^ | 12:54 |
| *** markmc has joined #openstack-infra | 12:55 | |
| *** adalbas has joined #openstack-infra | 12:55 | |
| *** xBsd has quit IRC | 12:55 | |
| openstackgerrit | Petr Blaho proposed a change to openstack-infra/config: Voting: false for py33, pylint for tuskarclient https://review.openstack.org/42632 | 12:56 |
| anteaya | chmouel: http://ci.openstack.org/stackforge.html | 12:57 |
| *** erfanian has joined #openstack-infra | 13:04 | |
| *** xBsd has joined #openstack-infra | 13:05 | |
| *** nayward has quit IRC | 13:10 | |
| xqueralt | hey, I think tempest tests might be broken for grizzly | 13:10 |
| xqueralt | could somebody take a look at: https://bugs.launchpad.net/tempest/+bug/1213912 ? | 13:11 |
| uvirtbot | Launchpad bug 1213912 in tempest "keystone cli tests fail in grizzly" [Undecided,New] | 13:11 |
| xqueralt | I've tried to debug it a bit more more but I don't know where the problem might come from | 13:12 |
| anteaya | hi xqueralt, tempest tests tend to be the purview of the folks in #openstack-qa, you might get more of a response to your question if you were to ask in that channel | 13:14 |
| xqueralt | anteaya ok, thanks | 13:15 |
| anteaya | np | 13:15 |
| *** lyz_ has joined #openstack-infra | 13:17 | |
| *** boris-42_ has joined #openstack-infra | 13:18 | |
| *** boris-42 has quit IRC | 13:20 | |
| *** boris-42_ is now known as boris-42 | 13:20 | |
| *** lyz has quit IRC | 13:22 | |
| *** stevebaker has quit IRC | 13:22 | |
| *** Guest63475 has quit IRC | 13:22 | |
| *** antonym has quit IRC | 13:22 | |
| *** oubiwann has quit IRC | 13:22 | |
| *** stevebaker has joined #openstack-infra | 13:23 | |
| *** burt has joined #openstack-infra | 13:26 | |
| *** Steely_Dan is now known as dansmith | 13:27 | |
| *** antonym has joined #openstack-infra | 13:27 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Separate reporters from triggers https://review.openstack.org/42643 | 13:28 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Move gerrit specific result actions to only apply to gerrit reporter https://review.openstack.org/42644 | 13:28 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for emailing results via SMTP https://review.openstack.org/42645 | 13:28 |
| *** erfanian has quit IRC | 13:29 | |
| *** vogxn has joined #openstack-infra | 13:29 | |
| *** oubiwann has joined #openstack-infra | 13:30 | |
| *** lillie has joined #openstack-infra | 13:32 | |
| *** lillie is now known as Guest75819 | 13:32 | |
| *** ryanpetrello has joined #openstack-infra | 13:32 | |
| *** psedlak has joined #openstack-infra | 13:36 | |
| * anteaya finishes reading weekend backscroll | 13:37 | |
| *** psedlak__ has quit IRC | 13:37 | |
| anteaya | a relatively quiet weekend, I see | 13:37 |
| anteaya | well done, all | 13:37 |
| *** dina_belova has joined #openstack-infra | 13:43 | |
| *** pabelanger has quit IRC | 13:43 | |
| *** dkliban has quit IRC | 13:44 | |
| *** erfanian has joined #openstack-infra | 13:45 | |
| *** dina_belova has quit IRC | 13:47 | |
| *** ruhe has quit IRC | 13:49 | |
| *** ruhe_ has joined #openstack-infra | 13:51 | |
| *** ruhe_ has quit IRC | 14:01 | |
| *** xBsd has quit IRC | 14:05 | |
| *** psedlak_ has joined #openstack-infra | 14:05 | |
| *** cppcabrera has joined #openstack-infra | 14:06 | |
| *** cppcabrera has left #openstack-infra | 14:08 | |
| *** psedlak has quit IRC | 14:09 | |
| *** annegentle has joined #openstack-infra | 14:10 | |
| *** mikal has quit IRC | 14:11 | |
| *** ruhe has joined #openstack-infra | 14:11 | |
| *** mikal has joined #openstack-infra | 14:12 | |
| *** rustlebee is now known as russellb | 14:13 | |
| *** blamar has joined #openstack-infra | 14:16 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Fixed sending notifications to #openstack-marconi https://review.openstack.org/42653 | 14:19 |
| *** rcleere has joined #openstack-infra | 14:20 | |
| *** changbl has quit IRC | 14:22 | |
| *** markmcclain has joined #openstack-infra | 14:25 | |
| *** xqueralt is now known as xqueralt-afk | 14:28 | |
| Alex_Gaynor | If anyone's got a few seconds ^ fixes IRC notifications for marconi | 14:28 |
| *** xqueralt-afk is now known as xqueralt | 14:28 | |
| *** rnirmal has joined #openstack-infra | 14:29 | |
| *** dkliban has joined #openstack-infra | 14:31 | |
| anteaya | reviewed | 14:32 |
| Alex_Gaynor | thanks | 14:32 |
| *** ftcjeff has joined #openstack-infra | 14:32 | |
| anteaya | np | 14:33 |
| *** fbo is now known as fbo_away | 14:33 | |
| pblaho | anyone who gets a free cycle please see https://review.openstack.org/#/c/42632/ - sets voting for some tuskarclient jobs | 14:35 |
| *** dkranz has joined #openstack-infra | 14:36 | |
| *** datsun180b has joined #openstack-infra | 14:40 | |
| *** fbo_away is now known as fbo | 14:40 | |
| openstackgerrit | Jason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore hacking validations with noqa https://review.openstack.org/41713 | 14:41 |
| *** cdevienne has joined #openstack-infra | 14:44 | |
| openstackgerrit | Jason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore hacking validations with noqa https://review.openstack.org/41713 | 14:46 |
| *** nayward has joined #openstack-infra | 14:50 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixed sending notifications to #openstack-marconi https://review.openstack.org/42653 | 14:54 |
| Alex_Gaynor | Does that automatically deploy, or does someone need to kick a thing? | 14:56 |
| jeblair | Alex_Gaynor: automagic | 14:57 |
| Alex_Gaynor | jeblair: should I assume from now on that that's true of everything under puppet? | 14:57 |
| jeblair | Alex_Gaynor: pretty much. :) we don't automatically restart services that would be particularly disruptive (eg zuul or gerrit), but otherwise, yes as much as possible | 14:58 |
| *** pabelanger has joined #openstack-infra | 14:59 | |
| openstackgerrit | Petr Blaho proposed a change to openstack-infra/config: Voting: false for py33, pylint for tuskarclient https://review.openstack.org/42632 | 15:01 |
| *** openstackgerrit has quit IRC | 15:01 | |
| *** mgagne has joined #openstack-infra | 15:02 | |
| *** openstackgerrit has joined #openstack-infra | 15:02 | |
| mgagne | thanks to whoever planned and renamed the project/repo for puppet-neutron =) | 15:03 |
| jeblair | mgagne: mordred did that | 15:03 |
| mordred | what did I do? | 15:03 |
| mordred | oh! yay! | 15:03 |
| openstackgerrit | Petr Blaho proposed a change to openstack-infra/config: Voting: false for py33, pylint for tuskarclient https://review.openstack.org/42632 | 15:03 |
| mgagne | mordred: thanks! now we can turn the page and go forward :D | 15:04 |
| jeblair | mordred: i reported bug 1213958 in novaclient | 15:06 |
| uvirtbot | Launchpad bug 1213958 in python-novaclient "UUID cache is not concurrency-safe" [Undecided,New] https://launchpad.net/bugs/1213958 | 15:06 |
| *** yaguang has joined #openstack-infra | 15:06 | |
| *** prad_ has joined #openstack-infra | 15:07 | |
| jeblair | because that's happening in nodepool | 15:08 |
| *** ruhe has quit IRC | 15:09 | |
| mordred | jeblair: awesome! | 15:11 |
| *** ruhe has joined #openstack-infra | 15:11 | |
| mordred | jog0: ^^ | 15:11 |
| *** psedlak_ has quit IRC | 15:11 | |
| jeblair | mordred: i wonder if heat has seen that? My vague understanding from a quick grep of the source is that heat uses a single globally cached Client (which is the model nodepool is using) | 15:12 |
| *** nayward has quit IRC | 15:13 | |
| mordred | jeblair: that's the model that you're using now to work around the concurrency issues, right? | 15:13 |
| mordred | jeblair: and I'm not sure if it is or not - SpamapS might know | 15:13 |
| *** nayward has joined #openstack-infra | 15:14 | |
| jeblair | mordred: the lingering file-descriptor issues; which I haven't quite tracked down enough to produce a good bug report (weird stuff started happening when I tried inspecting the object graph to find out what was going on) | 15:14 |
| jeblair | (inspecting the objects caused them to clean up. sigh) | 15:15 |
| mordred | wow. nice | 15:15 |
| mordred | jeblair: btw - I was having a call with a dude at HP today telling him about how all of our stuff works (he was impressed, fwiw) | 15:16 |
| *** thomasbiege1 has joined #openstack-infra | 15:16 | |
| mordred | jeblair: and had a thought that you might have already had... | 15:16 |
| mordred | jeblair: the code coverage jobs that we run in jenkins right now - should we have something that takes their output and spits it into graphite? | 15:17 |
| mordred | jeblair: so that we could have global report page thing | 15:17 |
| mordred | ? | 15:17 |
| mordred | (low priority to work on that, just wanted to say it out loud while the thought was in my head) | 15:17 |
| jeblair | mordred: sure, that would be a good idea i think. want to file a bug for it? | 15:18 |
| mordred | jeblair: sure thing | 15:19 |
| *** SergeyLukjanov has joined #openstack-infra | 15:19 | |
| *** changbl has joined #openstack-infra | 15:19 | |
| *** mrodden has quit IRC | 15:20 | |
| *** dina_belova has joined #openstack-infra | 15:21 | |
| jeblair | mordred: http://tinyurl.com/kmotmns | 15:21 |
| *** cdevienne has left #openstack-infra | 15:21 | |
| jeblair | mordred: also, we time out waiting for ssh access to hpcloud nodes _a lot_ | 15:21 |
| mordred | jog0: wow | 15:22 |
| mordred | jeblair: oh yeah? | 15:22 |
| mordred | like, the 600 second wait-for-node-to-really-be-ready timeout? | 15:22 |
| jeblair | mordred: 308 times in the 15.5 hours since utc midnight | 15:22 |
| mordred | wow | 15:22 |
| mordred | how many timeouts in that period? | 15:23 |
| jeblair | mordred: it's a 60 second timeout | 15:23 |
| jeblair | mordred: so it's _after_ the node becomes ACTIVE, it takes more than 60 seconds to establish an ssh connection | 15:24 |
| Alex_Gaynor | So, I want to get marconi running it's tests under pypy. To do this we need some buidlers with pypy! Where do I start to get PyPy installed under those machines (it'll be a PPA + apt-get to start with)? | 15:24 |
| mordred | jeblair: nod | 15:24 |
| *** ruhe has quit IRC | 15:24 | |
| SpamapS | jeblair: re using the globally cached nova client, I believe that is the case yes. | 15:25 |
| jeblair | SpamapS: is heat multithreaded, in particular such that multiple threads may use that single novaclient instance simultaneously? | 15:25 |
| *** ruhe has joined #openstack-infra | 15:28 | |
| mordred | jeblair: I have reported the timeouts to someone at HP and have copied Clark on the email | 15:28 |
| jeblair | Alex_Gaynor: i think we'd want a new slave definition (to keep it completely separate from other python unit tests) | 15:29 |
| Alex_Gaynor | jeblair: don't all the py2.6-3.3 ones run on the same slaves? | 15:30 |
| jeblair | Alex_Gaynor: you can probably pattern it after the work for the py3k slaves | 15:30 |
| Alex_Gaynor | jeblair: okey doke, where in teh repo are those? | 15:30 |
| jeblair | Alex_Gaynor: each version gets its own slave class | 15:30 |
| Alex_Gaynor | ah ok, sounds easy enough | 15:30 |
| * Alex_Gaynor has visions of spending 6 years learning puppet in a monestary in tibet | 15:30 | |
| jeblair | Alex_Gaynor: py26 has its own because centos6 is the only os that supports it... | 15:30 |
| jeblair | Alex_Gaynor: py3k has its own because the pip puppet module can't deal with multiple python installations | 15:31 |
| SpamapS | jeblair: no it is not multi-threaded, but it does use co-routines | 15:31 |
| SpamapS | jeblair: so there will likely be multiple nova requests going on at one time | 15:31 |
| jeblair | Alex_Gaynor: and for the moment (i believe mordred is working to change this) we do have to pip install some things globally (tox, and other stuff) | 15:32 |
| jeblair | Alex_Gaynor: (we want those to be packages) | 15:32 |
| mordred | yup | 15:33 |
| *** mrodden has joined #openstack-infra | 15:33 | |
| jeblair | Alex_Gaynor: so anyway, it's possible that pypy could co-exist on the py27 slaves; um, i wouldn't hold my breath though, i'm sure something will break that | 15:33 |
| mordred | not going to be fixed before I'm back from the desert though | 15:33 |
| Alex_Gaynor | jeblair: cool, where in teh repo should I poke? | 15:33 |
| jeblair | Alex_Gaynor: so i'd 'git grep py3k' and model the work after that. | 15:33 |
| Alex_Gaynor | jeblair: sounds good | 15:33 |
| jeblair | Alex_Gaynor: that even has an apt-repo that's only used for that slave | 15:33 |
| jeblair | class | 15:33 |
| *** vogxn has quit IRC | 15:34 | |
| HenryG | mordred: ping | 15:34 |
| *** vogxn has joined #openstack-infra | 15:34 | |
| markmcclain | clarkb: are you ok with me approving: https://review.openstack.org/#/c/35804/ | 15:35 |
| HenryG | mordred: OK if I take over https://review.openstack.org/35085 ? | 15:37 |
| mordred | HenryG: absolutely | 15:37 |
| *** boris-42 has quit IRC | 15:37 | |
| HenryG | mordred: thanks | 15:38 |
| *** boris-42 has joined #openstack-infra | 15:39 | |
| *** UtahDave has joined #openstack-infra | 15:41 | |
| jeblair | i'm going to manually clean up old images/nodes that leaked from the old devstack-gate scripts | 15:42 |
| mordred | jeblair: awesome | 15:42 |
| openstackgerrit | A change was merged to openstack/requirements: python3: Add mox3 dependency https://review.openstack.org/42493 | 15:42 |
| mordred | jeblair: " | 15:43 |
| mordred | Remember that ACTIVE in Nova just means that the VM boot process has | 15:43 |
| mordred | commenced." | 15:43 |
| mordred | jeblair: apparently, nova has status codes that are meaningless | 15:43 |
| jeblair | mordred: yeah, so basically it's a machine that's taking more than 60 seconds to boot. | 15:43 |
| mordred | yah | 15:43 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Delay 1 min before deleting node https://review.openstack.org/42531 | 15:43 |
| jeblair | mordred: we may not want to run tests on that machine. :) | 15:44 |
| mordred | :) | 15:44 |
| *** reed has joined #openstack-infra | 15:44 | |
| mordred | jeblair: do you happen to have success rate for that same period? | 15:44 |
| mordred | jeblair: and that's az1 and az2? or all three? | 15:45 |
| *** blamar_ has joined #openstack-infra | 15:45 | |
| jeblair | mordred: from midnight utc to now, fail: 369, success: 2150 | 15:45 |
| mordred | jeblair: cool. thanks | 15:45 |
| jeblair | so 17% | 15:45 |
| *** blamar has quit IRC | 15:45 | |
| *** blamar_ is now known as blamar | 15:45 | |
| jeblair | let me see if i can narrow az | 15:45 |
| jeblair | mordred: i'd have to write a script to tie 2 log lines together :( | 15:46 |
| reed | hello folks | 15:46 |
| jeblair | mordred: can do if you want, let's call it a 15 min job. | 15:47 |
| jeblair | reed: welcome back | 15:47 |
| mordred | jeblair: nah. I don't think it's necessary | 15:47 |
| mordred | jeblair: tanks | 15:47 |
| mordred | thanks | 15:47 |
| *** xqueralt has left #openstack-infra | 15:47 | |
| mordred | jeblair: but we're not using az3 yet, right? | 15:48 |
| jeblair | mordred: yes we are | 15:48 |
| *** pblaho has quit IRC | 15:50 | |
| *** ruhe has quit IRC | 15:53 | |
| mordred | oh. neat! | 15:54 |
| *** SergeyLu_ has joined #openstack-infra | 15:54 | |
| jeblair | mordred: we are currently at quota on hpcloud | 15:55 |
| jeblair | mordred: check out that graph i sent earlier | 15:55 |
| reed | adding openstack-infra to the activity board... not sure how to deal with forked projects like gerrit though | 15:56 |
| reed | first question: is gerrit the only fork there? | 15:56 |
| *** hub_cap has joined #openstack-infra | 15:56 | |
| pleia2 | good morning | 15:56 |
| mordred | jeblair: that is a lot of nodes | 15:56 |
| hub_cap | hey does the zuul.o.o page define the order in the queue? is it top to bottom on that page? /me is curious | 15:56 |
| *** SergeyLukjanov has quit IRC | 15:57 | |
| anteaya | morning pleia2 morning reed | 15:57 |
| jeblair | mordred: also that really didn't take long. | 15:57 |
| mordred | reed: so - patches that we've made to the fork are activity on openstack | 15:57 |
| jeblair | reed: if you examine changes in gerrit (as opposed to git history), you will have the openstack-related activity | 15:58 |
| mordred | reed: upstream commits will not have gerrit changes associated with them - so basically, if gerrit has a change for a thing, it's openstack activity | 15:58 |
| mordred | yeah - what jeblair said :) | 15:58 |
| reed | mordred, right ... I'll have them investigate | 15:58 |
| reed | thanks | 15:58 |
| mordred | reed: I promise, we're not trying to be difficult when we suggest that looking at git history for openstack is the wrong place for them to look | 15:58 |
| *** vogxn has quit IRC | 15:58 | |
| anteaya | hub_cap: my understanding of http://status.openstack.org/zuul/ is that the jobs at the top of the page are first in the queue and the line forms below them | 15:58 |
| mordred | the picture that gives is just MASSIVELY incomplete | 15:58 |
| anteaya | hub_cap: so yes, top to bottom on that page | 15:59 |
| hub_cap | ok sweet i assumed that anteaya but didnt have any concrete data behind it :) | 15:59 |
| hub_cap | <3 | 15:59 |
| anteaya | makes sense | 15:59 |
| anteaya | :D | 15:59 |
| *** ruhe has joined #openstack-infra | 15:59 | |
| jeblair | mordred: i'm going to spin up new long-running slaves. | 15:59 |
| mordred | jeblair: great! | 16:00 |
| anteaya | I have just watched them as the jobs progress and complete | 16:00 |
| jeblair | anteaya: gate jobs have precedence over post jobs which have precedence over check jobs | 16:00 |
| anteaya | there we go, more info - thanks jeblair | 16:00 |
| *** nayward has quit IRC | 16:01 | |
| jeblair | mordred: i was going to spin up 4 precise nodes. i think we need a lot more than that. i think i'll add 16 (double the current number) | 16:03 |
| mordred | jeblair: sounds good | 16:04 |
| reed | jeblair, is something like this line that differentiates openstack commits from other commits in gerrit repo? Change-Id: Ibfbb8bec245ae494a49cb905925fd2c4f34686f2 | 16:05 |
| *** thomasbiege1 has quit IRC | 16:06 | |
| mordred | reed: no, that line will also be in the commits we got from upstream gerrit | 16:06 |
| reed | no | 16:06 |
| mordred | because they also use gerrit to manage gerrit | 16:06 |
| reed | ohhh, now I re-read jeblair's suggestion and I think I misunderstood it before | 16:08 |
| *** ruhe has quit IRC | 16:12 | |
| *** hub_cap has left #openstack-infra | 16:13 | |
| reed | mordred, jeblair: what other forks are in openstack-infra? | 16:13 |
| *** SergeyLu_ has quit IRC | 16:14 | |
| mordred | reed: openstack-infra/lodgeit openstack-infra/meetbot openstack-infra/puppet-dashboard openstack-infra/puppet-vcsrepo | 16:14 |
| reed | mordred, I understand that github is the wrong place... I wish we could go back in time and not use it at all, not even for mirroring the stuff | 16:15 |
| mordred | reed: you know about git.openstack.org now right? | 16:15 |
| reed | mordred, when did that start" | 16:16 |
| jeblair | mordred, reed: i'm not sure we've quite announced it yet... but i think we're about ready to. seems to be working. | 16:16 |
| reed | ah | 16:16 |
| mordred | yah. I've been using it anytime I send a link to someone | 16:16 |
| reed | n-ah-e-c-eh-ah | 16:16 |
| *** boris-42 has quit IRC | 16:16 | |
| *** vogxn has joined #openstack-infra | 16:17 | |
| jeblair | reed: we're about to switch all of our testing, documentation, and any other references in -infra to use it. | 16:17 |
| jeblair | nodes precise17-32 are launching | 16:18 |
| reed | jeblair, sweet | 16:18 |
| *** cthulhup has joined #openstack-infra | 16:20 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Change hpcloud max-servers to 93 https://review.openstack.org/42675 | 16:21 |
| clarkb | morning | 16:22 |
| jeblair | clarkb: morning! you have to see this graph: http://tinyurl.com/kmotmns | 16:23 |
| clarkb | markmcclain: yes fine with me | 16:24 |
| jeblair | clarkb: i think we got within 2 servers of _actually_ exhausting our devstack node resources. | 16:24 |
| mordred | jeblair: could you explain the Used state to me? That's "node currently running a devstack test" right? | 16:24 |
| jeblair | mordred: yes | 16:24 |
| mordred | k. just making sure I was reading it right | 16:24 |
| clarkb | tldr a lot of tests :) | 16:25 |
| *** BobBall is now known as BobBall_Away | 16:25 | |
| clarkb | jeblair so 30 nodes seems about right | 16:25 |
| jeblair | clarkb: maybe? i still think it might be able to be lower | 16:26 |
| *** pabelanger has quit IRC | 16:26 | |
| jeblair | clarkb: when i said within 2 of exhaustion, i meant that we were at quota | 16:26 |
| clarkb | oh. there did appear to be a small sliver of green at one point | 16:26 |
| clarkb | I think we can live with a little node exhaustion while running 250 tests | 16:27 |
| jeblair | clarkb: but the pool size actually only needs to be big enough so that it doesn't hit 0 between when patches are uploaded and replacement servers are spun up | 16:27 |
| jeblair | clarkb: once we actually hit quota, it doesn't matter what the pool size is, it'll eventually hit 0 | 16:27 |
| *** nicedice_ has joined #openstack-infra | 16:28 | |
| jeblair | clarkb: so basically, we'd want to set it to some value where it doesn't hit 0 during the times that there is excess capacity. | 16:28 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add additional Rackspace region to nodepool config https://review.openstack.org/42677 | 16:29 |
| mordred | jeblair: think it's worth trying that? ^^ | 16:30 |
| *** markmc has quit IRC | 16:30 | |
| *** derekh has quit IRC | 16:31 | |
| clarkb | mordred I do, but defer to jeblair on actual deciding | 16:32 |
| mordred | me too | 16:32 |
| jeblair | mordred: trying: yes; throwing into production without any testing like that patch does: i'm not excited about that. | 16:32 |
| jeblair | mordred: i'd give it a new node-type, then we can create a job that runs on that node and try it out | 16:33 |
| mordred | I consider jeblair's brain to be absolute on all things | 16:33 |
| mordred | jeblair: ahhhh. see, that's cause you're smarter | 16:33 |
| * mordred goes to make that change | 16:33 | |
| *** sdake has joined #openstack-infra | 16:33 | |
| *** jpich has quit IRC | 16:34 | |
| * fungi agrees with everything said above, but is headed to the aquarium for a while and will catch up when he gets back | 16:36 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Try additional Rackspace region https://review.openstack.org/42677 | 16:37 |
| jeblair | mordred: how about we just manually trigger the job for now? if something goes wrong, i'm worried about getting a backlog in the silent queue | 16:39 |
| Alex_Gaynor | mordred: is it in the service catalog for the relevant users already? (I know it's a phased rollout) | 16:39 |
| jeblair | mordred: also needs a new image section under targets | 16:39 |
| openstackgerrit | Jason Meridth proposed a change to openstack-dev/hacking: Adds ability to ignore hacking validations with noqa https://review.openstack.org/41713 | 16:39 |
| mordred | jeblair: nod. so leave it out of zuul altogether? | 16:40 |
| jeblair | mordred: yeah. and maybe add the env variables like the periodic jobs have to make it easier to trigger | 16:40 |
| jeblair | Alex_Gaynor: i'll try some nova commands manually and see | 16:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Change hpcloud max-servers to 93 https://review.openstack.org/42675 | 16:42 |
| HenryG | I submitted https://review.openstack.org/42473 with no dependencies. Now I want to make it depend on https://review.openstack.org/35085 | 16:43 |
| HenryG | How do I do that? | 16:43 |
| *** vogxn has quit IRC | 16:43 | |
| mordred | HenryG: git review -d 35085 ; git review -x 42473 ; git review | 16:44 |
| jeblair | Alex_Gaynor, mordred: yeah, i can nova boot there | 16:44 |
| Alex_Gaynor | jeblair: awesome | 16:44 |
| *** jfriedly has joined #openstack-infra | 16:44 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Try additional Rackspace region https://review.openstack.org/42677 | 16:45 |
| mordred | jeblair: thanks - this has been a helpful exercise for me to use the nodepool yaml file (I usually learn by using) | 16:45 |
| *** tjones has joined #openstack-infra | 16:45 | |
| jeblair | mordred: lgtm. (some of my caution is also because i'm worried a little about bitrot; it's been a while since we've run a devstack job on rax) | 16:46 |
| mordred | jeblair: yah. excellent point | 16:46 |
| *** marun has joined #openstack-infra | 16:46 | |
| mordred | it would be super neat if we could start running there again | 16:46 |
| jeblair | ++ | 16:47 |
| *** ruhe has joined #openstack-infra | 16:48 | |
| * mordred run to bank - biab | 16:48 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add WSME to StackForge. https://review.openstack.org/42685 | 16:50 |
| jeblair | clarkb: did you want to review 42677? | 16:50 |
| clarkb | jeblair: sure | 16:50 |
| clarkb | doing that now | 16:50 |
| *** afazekas_ has joined #openstack-infra | 16:51 | |
| *** fbo is now known as fbo_away | 16:53 | |
| clarkb | jeblair: mordred: any concern with setting the total min ready nodes to 10? | 16:54 |
| *** ^d has joined #openstack-infra | 16:55 | |
| *** ^d has joined #openstack-infra | 16:55 | |
| *** yaguang has quit IRC | 16:55 | |
| jeblair | clarkb: well, none except that i'll probably have to manually clean them up, so maybe we ought to set it to 2-per master. | 16:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Try additional Rackspace region https://review.openstack.org/42677 | 16:57 |
| *** cthulhup has quit IRC | 16:58 | |
| jeblair | clarkb: got a few minutes for some jenkins clicking? | 16:59 |
| clarkb | jeblair: sure | 16:59 |
| clarkb | what are we clicking? | 17:00 |
| jeblair | clarkb: can you add the odd-numbered precise17-precise31 slaves to jenkins01? | 17:00 |
| clarkb | yup | 17:00 |
| clarkb | jeblair: they are already spun up? | 17:00 |
| jeblair | clarkb: did we decide that if we copied a jenkins node, it would immediately start an executor on the original? | 17:00 |
| jeblair | clarkb: yes, they are spun up | 17:01 |
| clarkb | jeblair: yes, copying is bad. I have been creating dumb slaves from scracth and letting my browser autofill stuff | 17:01 |
| jeblair | clarkb: k, thx | 17:01 |
| jeblair | i will do the evens | 17:01 |
| *** eharney has joined #openstack-infra | 17:01 | |
| *** ruhe has quit IRC | 17:02 | |
| clarkb | jeblair: though what are we doing about ssh credentials now? | 17:02 |
| clarkb | using jenkins (Jenkins SSH Key) ? | 17:03 |
| jeblair | clarkb: yep | 17:03 |
| mtreinish | jeblair, clarkb, fungi, mordred: the testr parallel runs seem to be taking a long time: https://jenkins02.openstack.org/job/gate-tempest-devstack-vm-testr-full/buildTimeTrend but the tempest run time is still the same | 17:03 |
| *** pabelanger has joined #openstack-infra | 17:03 | |
| mtreinish | do you think it was just a devstack change that made it slow, or could something else be going on? | 17:03 |
| jeblair | clarkb: hrm, i think i forgot to add the credential id to nodepool; i'll have to fix that in a bit | 17:04 |
| jeblair | mtreinish: can i get back to you in a few minutes? | 17:04 |
| mtreinish | jeblair: sure np | 17:05 |
| *** jerryz has joined #openstack-infra | 17:07 | |
| *** Ryan_Lane has joined #openstack-infra | 17:07 | |
| *** jerryz has quit IRC | 17:07 | |
| *** jerryz has joined #openstack-infra | 17:07 | |
| clarkb | jeblair: new nodes added to 01 | 17:09 |
| clarkb | precise27 isn't happy for some reason | 17:09 |
| Alex_Gaynor | jeblair: so I'm looking at the slave.pp, I assume this is the right place, basically what I think needs to happen for a pypy builder is just to have the pypy thing installed, and then make the `tox -epypy` variant run, it doesn't really matter what python tox itself is installed with | 17:09 |
| Alex_Gaynor | jeblair: am I off base? | 17:10 |
| clarkb | Alex_Gaynor: that sounds about right to me. With the one catch that tox doesn't always give you a super clean env (we have seen tox running under py27 leak py27 libs into py26 envs) | 17:10 |
| anteaya | hmmmm, I was about to change the link in https://wiki.openstack.org/wiki/ReviewChecklist from https://github.com/openstack-dev/hacking/blob/master/HACKING.rst to https://git.openstack.org/cgit/openstack-dev/hacking/tree/HACKING.rst but I see cgit doesn't render .rst files as .rst files | 17:10 |
| anteaya | is there a cgit plugin to get it to render .rst files as .rst files? | 17:11 |
| clarkb | jeblair: I can't ssh to precise27 either | 17:11 |
| anteaya | pleia2: do you know? | 17:11 |
| *** nati_ueno has joined #openstack-infra | 17:11 | |
| pleia2 | anteaya: there is not that I'm aware of | 17:11 |
| pleia2 | anteaya: looks like someone has done some work though: http://blog.zx2c4.com/325 | 17:11 |
| anteaya | hmmm, any thoughts on how we point folks to cgit .rst files and make them happy for clicking? | 17:12 |
| *** mezzovide has joined #openstack-infra | 17:12 | |
| *** ruhe has joined #openstack-infra | 17:12 | |
| clarkb | jeblair: perhaps that slave got caught in the ipv6 ssh nastiness and we deleted it? I am going to chck in the account now | 17:12 |
| *** dims has joined #openstack-infra | 17:13 | |
| jeblair | clarkb: hrm, i wouldn't have thought that launch-node would print the dns info in that case | 17:13 |
| clarkb | jeblair: the node is present and ACTIVE | 17:14 |
| clarkb | jeblair: and the A record for it matches what nova list shows | 17:14 |
| anteaya | pleia2: so basically this solution intercepts the server request for .rst files and converts them to .html files, rendering the .html, is that correct? | 17:14 |
| jeblair | clarkb: today would be a great day to have the git.o.o change merged. | 17:14 |
| jeblair | the load average on review.o.o is 229. | 17:14 |
| clarkb | jeblair: wow | 17:14 |
| anteaya | pleia2: do we have an .html file as a sample anywhere in any of our repos? | 17:15 |
| clarkb | jeblair: I will put the git.o.o change at the top of my review list | 17:15 |
| clarkb | jeblair: do you want me to keep looking at precise27 or should we just ignore that for now? | 17:15 |
| jeblair | i'm watching a job on a new node, and it's spending a lot of time cloning | 17:15 |
| jeblair | clarkb: why don't you finish looking into precise27 first | 17:16 |
| jeblair | clarkb: don't spend long on it; if you decide it's a lost cause, let me know and i'll recreate it (i still have a window open and commands handy) | 17:16 |
| *** colinmcnamara has joined #openstack-infra | 17:16 | |
| *** pentameter has joined #openstack-infra | 17:16 | |
| clarkb | jeblair: it unpings, ssh cannot get to it but nova list reports it is active. I could get on the web console, but deleting it and trying again seems simpler | 17:17 |
| clarkb | especially since we haven't really invested anything in the current node | 17:17 |
| *** burt has quit IRC | 17:18 | |
| jeblair | clarkb: ok will do. i'm about to upload a new patchset of the git change | 17:18 |
| pleia2 | anteaya: I'm sure we do, looking | 17:18 |
| anteaya | pleia2: if I read this correctly, cgit 0.9.2 has the ability to convert .md/.rst/.txt files converted for output: http://www.phoronix.com/scan.php?page=news_item&px=MTM3OTY | 17:19 |
| Alex_Gaynor | clarkb: So http://bpaste.net/show/124263/ is what I've got so far, can you let me know if I'm on the right track? | 17:19 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use cgit server instead of github for everything https://review.openstack.org/38177 | 17:19 |
| clarkb | Alex_Gaynor: sure, let me review 38177 first really quick | 17:19 |
| jeblair | clarkb: okay, let's review that now ^ | 17:19 |
| Alex_Gaynor | clarkb: sure thing, thanks | 17:20 |
| *** arosen has left #openstack-infra | 17:20 | |
| *** arezadr has joined #openstack-infra | 17:20 | |
| pleia2 | anteaya: so I noticed it would do it for some README files, let me dig in a bit | 17:20 |
| clarkb | jeblair: and I am going to review locally as you can really tell gerrit is struggling | 17:21 |
| *** mezzovide has quit IRC | 17:21 | |
| jeblair | clarkb: a lot (most) of the load is apache serving git | 17:21 |
| clarkb | jeblair: I have a feeling we should grow git.o.o if that is the case | 17:22 |
| anteaya | pleia2: yeah when I go to the cgit home page it looks like some form of markup is at play: http://git.zx2c4.com/cgit/about/ | 17:22 |
| mordred | anteaya, jog0: you know - we might want to make the hacking guidelines a page in sphinx docs for the hacking repo | 17:22 |
| clarkb | jeblair: and probably before we switch everything to it (git.o.o is a 1GB node and gerrit is 32GB, if apache is most of the load ... | 17:22 |
| anteaya | mordred: not a bad idea, what do you think jog0? | 17:22 |
| jeblair | 1g? uh wow. | 17:23 |
| *** pcrews has joined #openstack-infra | 17:23 | |
| mordred | anteaya, jog0: and then publish to docs.openstack.org/developer/hacking | 17:23 |
| clarkb | jeblair: or spin up a few of them and load balance | 17:24 |
| jeblair | clarkb: we can also scale it horizontally. so perhaps what we want to do is grow this one to 16 or 32; check out the load, and then add a few smaller ones and then delete the current one. | 17:24 |
| mordred | pleia2, anteaya: and in general, probably if it's a thing we want to point someone to a link to read - we should probably think of it in terms of what sphinx repo it can live in - rather than linking to magical RST things in git repos. I think. I might be wrong | 17:24 |
| clarkb | jeblair: ++ | 17:24 |
| mordred | yay. more 38177 | 17:25 |
| *** pblaho has joined #openstack-infra | 17:26 | |
| anteaya | mordred: fair enough regarding links. I do think that digging into how to get cgit to render .rst files is a good short term use of time, given we have so many of them since we had been going that direction | 17:26 |
| mordred | anteaya: totally | 17:26 |
| jeblair | clarkb: i'll grow the current git.o.o to 30gb? it might take a while, but that's what it takes to get 8vcpus. | 17:26 |
| anteaya | but introducing a stronger push to creating sphinx docs is a good push to have, I agree | 17:26 |
| Alex_Gaynor | remember for a spihinx thing, it's not just rendering a .rst file, it's the full sphinx system | 17:26 |
| pleia2 | anteaya: yeah, so it seems like in this version it's really strictly tied to about pages | 17:26 |
| mordred | when we're done with 38177, and we move on to thinking about cloning/pulling from git.o.o instead of from review.o.o - should we also consider replacing the gitweb links in gerrit with git.o.o links? | 17:27 |
| pleia2 | not random-rst-files in the repo | 17:27 |
| anteaya | pleia2: huh, can you find the url for the cgit/about page? so far I have been unable | 17:27 |
| jeblair | mordred: yes | 17:27 |
| mordred | jeblair: cool | 17:27 |
| clarkb | jeblair: wfm. I think without growing it or scaling horizontally is something we need to do in any case. so waiting is fine | 17:27 |
| * dhellmann perks up at the mention of rst | 17:27 | |
| dhellmann | mordred, anteaya : the rtd.org code is opensource | 17:27 |
| pleia2 | anteaya: not sure what you mean | 17:28 |
| anteaya | pleia2: hmmmm, so a stronger push for sphinx it feels like | 17:28 |
| mordred | dhellmann: yes it is | 17:28 |
| anteaya | hey dhellmann | 17:28 |
| *** olaph has quit IRC | 17:28 | |
| *** afazekas is now known as afazekas_zz | 17:28 | |
| jeblair | resize in progress | 17:28 |
| pleia2 | anteaya: the template for cgit/about or something on our specific install..? | 17:28 |
| pleia2 | (we currently don't have a repo about page enabled) | 17:28 |
| anteaya | pleia2: when I try to find the source code for the cgit about page I look in the cgit/tree and can not find the file marked about: http://git.zx2c4.com/cgit/tree/ | 17:28 |
| pleia2 | ah, gotcha | 17:29 |
| dhellmann | mordred, anteaya : but Alex_Gaynor is right that if we're just rendering one-off rst files, rst2html from docutils is probably simpler | 17:29 |
| Alex_Gaynor | dhellmann: but also insufficient for things that are actually sphinx | 17:29 |
| mordred | yah. | 17:29 |
| dhellmann | Alex_Gaynor: yep | 17:29 |
| *** ruhe has quit IRC | 17:29 | |
| pleia2 | anteaya: I don't think it's included in the package, up to the admin to set up a root-readme=/var/www/html/about.html | 17:29 |
| mordred | so - I think that it seems like the hacking guidelines really are what the sphinx docs of hacking should be in this particular case | 17:29 |
| mordred | but I agree that rst2html for simpler/one-off things also seems correct | 17:30 |
| anteaya | dhellmann Alex_Gaynor any opinion on whether it is worthwhile to pursue this route as an .rst to .html fix for us? http://blog.zx2c4.com/325 | 17:30 |
| dhellmann | that makes sense | 17:30 |
| anteaya | pleia2: ah, a behind the scenes about file | 17:30 |
| Alex_Gaynor | anteaya: is the usecase our sphinx docs? | 17:30 |
| anteaya | so we wouldn't be going that way | 17:30 |
| pleia2 | anteaya: well, we could commit one to /config ourselves, currently dn't yet have the root-readme option enabled in cgitrc | 17:31 |
| anteaya | the thread I pulled that has unraveled is the link in the review guidelines wiki: https://wiki.openstack.org/wiki/ReviewChecklist | 17:31 |
| dhellmann | anteaya: as Alex_Gaynor pointed out, that's going to work for the subset of rst that docutils understands natively, but not any of the sphinx extensions | 17:31 |
| *** olaph has joined #openstack-infra | 17:31 | |
| anteaya | it links to the hacking.rst page on github (very pretty): https://github.com/openstack-dev/hacking/blob/master/HACKING.rst | 17:31 |
| anteaya | we are using cgit and I thought it would be a good idea to change the link | 17:32 |
| Alex_Gaynor | for HACKING.rst and other simple documents it's probably worth it | 17:32 |
| anteaya | https://git.openstack.org/cgit/openstack-dev/hacking/tree/HACKING.rst | 17:32 |
| jeblair | clarkb: precise27 deleted and rebuilding | 17:32 |
| anteaya | but .rst files in cgit render as .txt files unless they are an about page | 17:32 |
| clarkb | jeblair: 38177 lgtm. (waiting on gerrit to post my vote). I do think we may need to manually change the origin remotes on any puppet vcsrepo repos that change, other than that we should be good | 17:33 |
| anteaya | okay so short term, get current .rst files rendering nicely in cgit with this hack: http://blog.zx2c4.com/325 | 17:33 |
| jeblair | clarkb: yeah, we still have that 'set origin' command in the test scripts from a while ago, right? | 17:34 |
| anteaya | long term, create doc files via sphinx for any documentation and link wiki pages to sphinx generated docs | 17:34 |
| anteaya | pleia2 Alex_Gaynor dhellmann how do you feel about these two steps? | 17:34 |
| clarkb | jeblair: maybe? | 17:35 |
| anteaya | and mordred | 17:35 |
| jeblair | clarkb, mordred: we are now using our full devstack-gate capacity | 17:35 |
| mordred | jeblair: nice | 17:35 |
| dhellmann | anteaya: makes sense, maybe fallback to plaintext if there is any error rendering? | 17:36 |
| anteaya | dhellmann: I like that | 17:36 |
| Alex_Gaynor | clarkb, jeblair: A version with a few more things I think are needed: http://bpaste.net/show/124269/ (is there documentation on how to do this, I'm cargo culting like mad :)) | 17:36 |
| pleia2 | anteaya: works for me, we'll want to confirm that the .rst thing isn't too hokey - yeah, failback would be nice | 17:36 |
| clarkb | Alex_Gaynor: there is documentation for adding new jobs to zuul and JJB | 17:37 |
| anteaya | pleia2: okay I can work on a patch, with your help and in the meantime we can confirm the non-hokeiness of the proposal | 17:37 |
| anteaya | so by the time it is ready to merge we feel it is the right thing to do, yeah? | 17:37 |
| clarkb | Alex_Gaynor: but less so for puppet changes as I guess we assume some familiarity with it. You are on the right track. But our puppet is made up of onion layers like ogres. So you will need to pass include_pypi all the way from openstack_project::slave to jenkins::slave | 17:38 |
| *** thomasbiege1 has joined #openstack-infra | 17:38 | |
| pleia2 | anteaya: perfect, once you have a patch I can toss it up on my test instance and throw some things at it | 17:38 |
| anteaya | yay | 17:38 |
| clarkb | Alex_Gaynor: I would also give it a default value of false so that we don't have to set it to false for the centos slaves | 17:38 |
| anteaya | pleia2: I'll make you a nice target | 17:38 |
| pleia2 | :) | 17:38 |
| jeblair | clarkb: hrm, no gerrit-git-prep sets the origin to gerrit | 17:38 |
| jeblair | clarkb, mordred: i'll work on a followup that sets origin to git.o.o | 17:39 |
| clarkb | jeblair: ok | 17:39 |
| mordred | jeblair: k. | 17:39 |
| mordred | jeblair: I've got a patch coming up for the gitweb links | 17:39 |
| clarkb | Alex_Gaynor: I would also move the ppa section into the conditional block for debian derivatives in jenkins::slave | 17:39 |
| Alex_Gaynor | clarkb: http://bpaste.net/show/124274/ | 17:42 |
| pleia2 | oh btw, I have to run out for a bit around 2 (have a guest in town, said we'd do some lunch adventures) | 17:43 |
| *** moted has quit IRC | 17:43 | |
| mordred | when did linus start writing text editors? | 17:43 |
| mordred | https://git.kernel.org/cgit/editors/uemacs/uemacs.git/ ? | 17:43 |
| *** moted has joined #openstack-infra | 17:43 | |
| clarkb | mordred: he has been a long time user of that emacs clone thing iirc | 17:43 |
| clarkb | microemacs | 17:43 |
| anteaya | pleia2: I'm busy at 2 as well, so happy running around | 17:44 |
| *** nayward has joined #openstack-infra | 17:44 | |
| Alex_Gaynor | clarkb: Would this be easier if I did it via a WIP patch review? | 17:45 |
| clarkb | Alex_Gaynor: I want inline commenting. The zuul and JJB stuff looks just about there | 17:45 |
| anteaya | pleia2: am I in the right neck of the woods for a cgit patch? https://git.openstack.org/cgit/openstack-infra/config/tree/modules/cgit | 17:45 |
| Alex_Gaynor | clarkb: Ok, so I'll put this into gerrit then | 17:45 |
| clarkb | Alex_Gaynor: ya I think we are getting close enough to start refining in gerrit | 17:45 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Have gerrit-git-prep use git.o.o https://review.openstack.org/42691 | 17:46 |
| pleia2 | anteaya: I just updated http://ci.openstack.org/git.html ! :) | 17:46 |
| *** thomasbiege2 has joined #openstack-infra | 17:46 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi https://review.openstack.org/42692 | 17:46 |
| jeblair | clarkb, mordred: it turns out having our tests use git.o.o is completely separate from mordred's change... | 17:47 |
| pleia2 | anteaya: so yes, that dir, and then some in modules/openstack_project/manifests/git.pp and files/git/cgitrc | 17:47 |
| Alex_Gaynor | clarkb: ^ | 17:47 |
| anteaya | pleia2: fab thanks | 17:47 |
| *** melwitt has joined #openstack-infra | 17:47 | |
| jeblair | clarkb, mordred: one is s/github/ the other is s/review/. | 17:47 |
| anteaya | perfect, I will have a dig | 17:47 |
| jeblair | clarkb, mordred: and devstack-gate needs updating too. | 17:47 |
| mordred | jeblair: yah | 17:49 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use git.openstack.org as origin https://review.openstack.org/42693 | 17:49 |
| *** olaph has quit IRC | 17:49 | |
| jeblair | clarkb, mordred: those 2 changes should do it ^ | 17:49 |
| *** thomasbiege1 has quit IRC | 17:50 | |
| mordred | first lgtm | 17:50 |
| *** pabelanger has quit IRC | 17:50 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make the gitweb links in gerrit point to git.o.o https://review.openstack.org/42694 | 17:51 |
| *** dina_belova has quit IRC | 17:51 | |
| *** afazekas_ has quit IRC | 17:51 | |
| anteaya | pleia2: 3 links on this page go to github: http://ci.openstack.org/git.html 2 puppet links and 1 configuration link | 17:51 |
| jeblair | mordred: next time someone asks you how many jenkins slaves we have, you tell them 328. | 17:51 |
| mordred | jeblair: HAHAHAHA. nice | 17:51 |
| clarkb | I think I managed to successfully put a +2 on the first from the command line | 17:51 |
| clarkb | jeblair: is that all? | 17:52 |
| jeblair | and counting | 17:52 |
| *** olaph has joined #openstack-infra | 17:52 | |
| anteaya | pleia2: btw, can I edit this page? it looks like a sphinx generated doc page | 17:52 |
| mordred | I have +2'd both | 17:52 |
| Alex_Gaynor | clarkb: I'm in disbelief that I managed to get a maybe working thing on my first try. | 17:52 |
| pleia2 | anteaya: yeah, we haven't moved our docs over to using git.o.o yet (there are a couple patches), I need to add this one to patches too | 17:52 |
| clarkb | Alex_Gaynor: I don't think it is quite working, it is close though | 17:53 |
| pleia2 | anteaya: it's in config/doc/source | 17:53 |
| Alex_Gaynor | clarkb: hehe | 17:53 |
| jeblair | mordred, clarkb: i think we want the resize to finish before merging my changes (probably yours too) | 17:53 |
| clarkb | jeblair: yes | 17:53 |
| mordred | jeblair: yah | 17:53 |
| anteaya | pleia2: ah okay, so I will hold off until your patch goes through, yes? | 17:53 |
| pleia2 | anteaya: ah, good, this patch includes git https://review.openstack.org/#/c/38177 | 17:53 |
| pleia2 | so once that lands, our docs will be good :) | 17:54 |
| mordred | jeblair: sometimes I really don't think people understand exactly how many servers it takes to run our little project here | 17:54 |
| mordred | you'd think to yourself "oh, well, I mean, serving git repos must be easy - it's git" | 17:54 |
| pleia2 | anteaya: yeah, probably want to wait on doc edits until 38177 lands | 17:55 |
| pleia2 | should be soon though | 17:55 |
| anteaya | okay, it is in the monster do-everything patch, I will review | 17:55 |
| jeblair | um. git.opo has been resized to 8192, which is weird because i'm pretty sure i told it to do 30g. | 17:55 |
| *** odyssey4me3 has quit IRC | 17:55 | |
| mordred | jeblair: neat! it's great when they resize it to a different size | 17:55 |
| pleia2 | oh - we're also not doing any caching at the moment | 17:56 |
| clarkb | mordred: jeblair: should gitweb and cgit be an XOR? or are we fine potenitally running both? | 17:56 |
| *** pabelanger has joined #openstack-infra | 17:56 | |
| pleia2 | there are lots of caching options you can work with in the cgitrc: http://git.zx2c4.com/cgit/tree/cgitrc.5.txt | 17:56 |
| jeblair | i'm going to confirm the resize and then upgrade again from there (otherwise we'll have stale git repos if i revert it) | 17:56 |
| *** thomasbiege2 has quit IRC | 17:56 | |
| clarkb | jeblair: wfm | 17:57 |
| jeblair | (and i don't want to do a full replication push) | 17:57 |
| pleia2 | so we should figure out what we want as far as caching options, the default "cache-size=1000" in centos was unpleasant for testing | 17:57 |
| clarkb | pleia2: I was going to play iwth adding some cache stuff | 17:57 |
| pleia2 | clarkb: cool | 17:57 |
| clarkb | pleia2: was it the size of the cache or the ttl on the cahce items? | 17:57 |
| pleia2 | clarkb: number of cached items, whatever that means :) | 17:57 |
| pleia2 | "output entries" | 17:58 |
| clarkb | pleia2: I think we can reduce the ttl of the more dynamic stuff like log listings and so on and increase the cache ttl for static things like sha objects | 17:58 |
| clarkb | s/sha/commit/ | 17:58 |
| pleia2 | clarkb: yeah, like the front page we can probably set cache reset to a couple minutes or more | 17:58 |
| pleia2 | cache-root-ttl:: | 17:58 |
| Alex_Gaynor | clarkb: given the lack of confidence that htis 100% works, should I put these builds into non-voting to start with, and if so, where do I do that? | 17:59 |
| clarkb | Alex_Gaynor: I see I voted on the wrong change woot | 17:59 |
| clarkb | Alex_Gaynor: sorry for the confusing that +2 was meant for jeblairs change | 18:00 |
| Alex_Gaynor | clarkb: aaaaah, this addresses my confusion | 18:00 |
| *** dina_belova has joined #openstack-infra | 18:00 | |
| *** colinmcnamara has quit IRC | 18:00 | |
| clarkb | jeblair's change is 42693... | 18:00 |
| clarkb | this is what I get for trying to change my workflow to not bother review.o.o apache so much | 18:01 |
| *** boris-42 has joined #openstack-infra | 18:02 | |
| *** thomasbiege has joined #openstack-infra | 18:04 | |
| *** morganfainberg|a is now known as morganfainberg | 18:04 | |
| jeblair | mtreinish: looking | 18:05 |
| clarkb | Alex_Gaynor: your review has been udpated | 18:06 |
| mtreinish | jeblair: ok thanks | 18:06 |
| clarkb | too much excitement for monday but I guess that is what we get during freeze week | 18:06 |
| jeblair | mtreinish: looking at https://jenkins02.openstack.org/job/gate-tempest-devstack-vm-testr-full/2024/consoleFull | 18:06 |
| jeblair | mtreinish: it spent 40 minutes inside the wrap script. i don't have detailed timestamps for that, but my suspicion is that most of it was spent doing git operations | 18:07 |
| jeblair | mtreinish: which, interestingly enough, is what we're working on speeding up right now. :) | 18:08 |
| *** pabelanger has quit IRC | 18:08 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi https://review.openstack.org/42692 | 18:09 |
| mtreinish | jeblair: nice | 18:09 |
| mtreinish | ok, yeah 40 min is a long time for that, but that adds up well with the 20-30min for tempest in parallel :) | 18:09 |
| jeblair | dtroyer: how does devstack get those nifty bash timestamps? | 18:11 |
| clarkb | Alex_Gaynor: so close. One JJB quirk and one puppet bug to work around then I believe that change is good | 18:13 |
| jeblair | yay we're about to get more slaves :) | 18:13 |
| Alex_Gaynor | clarkb: coolio, will have a new version up in a few minutes, just need to finish writing an eamil | 18:13 |
| clarkb | jeblair: hmm? | 18:14 |
| jeblair | clarkb: pypy slaves :) | 18:14 |
| jeblair | clarkb: 328++ | 18:14 |
| *** thomasbiege has quit IRC | 18:15 | |
| jeblair | oh, i think i just found the thing in devstack that does the timestamps. | 18:15 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi https://review.openstack.org/42692 | 18:16 |
| *** jswarren has joined #openstack-infra | 18:16 | |
| jeblair | Alex_Gaynor: those should probably be two separate changes, otherwise we'll get a queue backlog while we build the hosts | 18:17 |
| *** rihnapstor has joined #openstack-infra | 18:17 | |
| jeblair | Alex_Gaynor: (which, sadly, at the moment is not automatic, but maybe soon) | 18:17 |
| *** colinmcnamara has joined #openstack-infra | 18:18 | |
| Alex_Gaynor | jeblair: sorry, what's not automatic? | 18:18 |
| jeblair | Alex_Gaynor: creating a new jenkins slave | 18:18 |
| clarkb | jeblair: it isn't a new slave. it is overloading the py3k slaves | 18:18 |
| jeblair | oh, i see | 18:18 |
| jeblair | perhaps i should actually read it | 18:19 |
| *** colinmcnamara has quit IRC | 18:19 | |
| *** vipul is now known as vipul-away | 18:21 | |
| *** thomasbiege has joined #openstack-infra | 18:23 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi https://review.openstack.org/42692 | 18:23 |
| Alex_Gaynor | lifeless: so trying to run any of the testr projects with tox -epypy results in "memoryview has no attribute 'decode'", which is a bit bizzaro to me, since it doesn't on CPython either, there's no traceback, any suggestions on how to debug? | 18:24 |
| *** thomasbiege has quit IRC | 18:24 | |
| *** SergeyLukjanov has joined #openstack-infra | 18:28 | |
| dtroyer | jeblair: context? in the logs? | 18:30 |
| jswarren | Seeing several Jenkins failures in seemingly unrelated changes..for nova, keystone and glance. | 18:31 |
| jeblair | dtroyer: yeah, i found it; the "exec 1>" bits | 18:31 |
| jswarren | Errors running gate-grenade-devstack-vm | 18:31 |
| jswarren | Several instances of "ERROR:root:Could not find any typelib for GnomeKeyring" | 18:32 |
| dtroyer | jeblair: yup. | 18:32 |
| jswarren | And "glance: error: unrecognized arguments: --public" | 18:33 |
| jswarren | Examples: https://review.openstack.org/#/c/41573/ and https://review.openstack.org/#/c/40232/ | 18:33 |
| *** lbragstad has joined #openstack-infra | 18:34 | |
| *** olaph has quit IRC | 18:35 | |
| dtroyer | jswarren: the —public thing is because they removed the —public option without thinking anyone used it…no backward compatibility I guess? | 18:35 |
| *** xBsd has joined #openstack-infra | 18:36 | |
| jswarren | Hmm. How could such a change get merged if Jenkins fails? | 18:36 |
| dtroyer | jswarren: the clients are supposed to maintain that as they are supposed to work all the way back to Essex I think is what we agreed to a while back. Grenade/devstack pulls the current clients as they are not released in the same cycle | 18:36 |
| dtroyer | jswarren: it got merged because trunk devstack doesn't use it | 18:36 |
| dtroyer | but in grenade both versions use current clients. we may need to change that, but also, this is catching breakage that shouldn't happen | 18:37 |
| jeblair | dtroyer: did we not run a grenade job for that change? | 18:37 |
| *** burt has joined #openstack-infra | 18:37 | |
| dtroyer | jeblair: I don' know, I haven't gone looking for it. but in general we don't run grenade on clients do we? | 18:37 |
| jeblair | dtroyer: we should; it is configured for python-glanceclient | 18:38 |
| *** vipul-away is now known as vipul | 18:39 | |
| dtroyer | jeblair: ok, I hadn't looked at them in a while | 18:39 |
| dtroyer | It's this change: https://review.openstack.org/39324 and no grenade run | 18:39 |
| jeblair | gate-grenade-devstack-vm is there | 18:40 |
| dtroyer | doh, it is | 18:40 |
| *** tjones has quit IRC | 18:40 | |
| *** olaph has joined #openstack-infra | 18:40 | |
| dtroyer | hmmm…I need to look at this…the base run clearly worked with —public | 18:41 |
| dtroyer | ah, glacneclient is installed from pip as part of the prereqs. devstack stable/grizzly is still the old borked way for deps... | 18:42 |
| ttx | mordred: still playing with the merge-mp-back-to-master on hte swift m-p branch ? Or can I delete it now ? | 18:42 |
| jeblair | dtroyer: weird, it should be installed from source first | 18:45 |
| dtroyer | so chicken-n-egg here…original grenade run uses glanceclient with —public still present…then test without —public under devstack that doesn't use it…it passes and is merged and now gets used in further tests with stable/grizzly devstack that wants —public. | 18:45 |
| jeblair | dtroyer: as i read it, it installed 0.10.0.7.g7e4ba22 for the old run and 0.10.0.9.g6c84bf3 for the new (i don't know why those are different) | 18:46 |
| clarkb | jeblair: how is the second resize going? | 18:46 |
| jeblair | clarkb: it looks to be done! | 18:46 |
| dtroyer | jeblair: the long-and-short is they broke cli backward compatibility | 18:46 |
| clarkb | cpuinfo and free confirm the new hardware | 18:46 |
| jeblair | dtroyer: yeah, they should probably revert that patch | 18:47 |
| jeblair | dtroyer: but i still think it shouldn't have landed | 18:47 |
| clarkb | I am going to grab lunch shortly, not sure if we want to merge the git.o.o cutover changes now or when we can all help debug | 18:47 |
| *** olaph has quit IRC | 18:47 | |
| dtroyer | jeblair: I see why it did and it's inherent in how grenade is built. | 18:49 |
| dtroyer | devstack was changed in https://review.openstack.org/39323, frankly that's how it should work anyway using the documented options. but that's why it got in | 18:52 |
| * clarkb lunches. feel free to merge the git.o.o switch stuff, but I won't be able to help updating remotes for about an hour | 18:53 | |
| clarkb | though the g-g-p change is probably most useful | 18:53 |
| *** nayward has quit IRC | 18:53 | |
| *** mrmartin has joined #openstack-infra | 18:54 | |
| mrmartin | hi | 18:55 |
| jeblair | dtroyer: yes, that makes sense. i think it installed the old version of glanceclient for the grizzly setup (actually, it would be nice if it didn't do that for the clients, but i think we actually need to test that with a different test), and then installed the proposed change for the new test | 18:55 |
| *** mriedem has quit IRC | 18:55 | |
| jeblair | dtroyer: which as you say, didn't exercise that option | 18:55 |
| *** mriedem has joined #openstack-infra | 18:56 | |
| jeblair | dtroyer: (i think the new job we need is 'test proposed changes to clients against stable branch devstacks') | 18:56 |
| jeblair | dtroyer: so i think we need to revert the commit or backport the option change to devstack stable branches, yeah? | 18:57 |
| mrmartin | jeblair: hi, could you help me with this review: https://review.openstack.org/#/c/42608/ | 18:57 |
| jeblair | clarkb: yes, let's do it after lunch | 18:57 |
| dtroyer | jeblair: the only problem there is the false security as theat is far from complete test, it'll only say that devstack jobs are likely to complete…which may be enough | 18:57 |
| *** ruhe has joined #openstack-infra | 18:58 | |
| jeblair | dtroyer: you're right, the proper solution is a tempest environment that runs cli tests with proposed changes to the cli | 18:58 |
| jeblair | mrmartin: sure | 18:59 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add WSME to StackForge. https://review.openstack.org/42685 | 18:59 |
| jeblair | mrmartin: what's your question? | 19:00 |
| mrmartin | jeblair: I got a failure with puppet-lint: http://logs.openstack.org/08/42608/2/check/gate-config-puppet-lint/3146b05/console.html.gz , does it mean, that a warning will result in Failure? | 19:00 |
| *** olaph has joined #openstack-infra | 19:00 | |
| jeblair | mrmartin: yes, our puppet-lint is MEAN. if you follow that link, and then the console log, you'll see it points out the 3 warnings about => alignment | 19:01 |
| jeblair | mrmartin: it's a bit hard to read, but the lines like "2013-08-19 12:31:03.391 | manifests/site.pp - WARNING: indentation of => is not properly aligned on line 158" | 19:01 |
| jeblair | mrmartin: are pretty much the only ones that matter in that output | 19:01 |
| mrmartin | yeah I see it, and this is not a problem for me, but if you check the previous one: http://logs.openstack.org/08/42608/1/check/gate-config-puppet-lint/3f135b6/console.html.gz | 19:01 |
| mrmartin | it was going through all files, and displayed the hints for them | 19:02 |
| jeblair | mrmartin: oh, did you not fix all of those between the two patchsets? | 19:02 |
| mrmartin | but for second run, this just show warning fors manifests/site.pp | 19:02 |
| mrmartin | jeblair, I fixed most of the warnings, but I forgot the site.pp, and I have >80length issues with another file | 19:03 |
| mgagne | jeblair: I think we should redirect stderr to stdout for puppet-lint, this would avoid the output from being mixed up (rake aborted! in between the error output) | 19:04 |
| mrmartin | I got that I need to repair those, it was just strange for me, that the first run checked all the files, and the second one just displayed error for site.pp | 19:04 |
| mordred | yikes. scrollback | 19:04 |
| SlickNik | reed: got a minute? | 19:04 |
| jeblair | mgagne: sounds like a good idea | 19:04 |
| mgagne | jeblair: I'll take care of that | 19:05 |
| reed | SlickNik, not really, heading out for lunch... | 19:05 |
| *** w_ has joined #openstack-infra | 19:05 | |
| jeblair | mrmartin: if there are supposed to be other warnings, i don't know why it wouldn't show up there. maybe a bug? it should run on all the files all the time | 19:05 |
| SlickNik | okay, will ping you later after lunch. | 19:05 |
| dtroyer | jeblair: I proposed a glanceclient revert: https://review.openstack.org/#/c/42706, let's see what kind of argument we can stir up… ;) | 19:05 |
| SlickNik | reed: okay, bon apetit, will ping you later. :) | 19:05 |
| *** nati_uen_ has joined #openstack-infra | 19:06 | |
| mrmartin | jeblair: ok, I'll repair the site.pp and push the patch again, let's see the result. | 19:06 |
| *** olaph has quit IRC | 19:08 | |
| *** nati_ueno has quit IRC | 19:08 | |
| *** ruhe has quit IRC | 19:09 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Redirect stderr to stdout for puppet-lint https://review.openstack.org/42709 | 19:10 |
| *** olaph has joined #openstack-infra | 19:11 | |
| *** w_ has quit IRC | 19:11 | |
| *** rihnapstor has quit IRC | 19:14 | |
| *** olaph has quit IRC | 19:19 | |
| ekarlso- | jenkins is utterly slow today or ? | 19:21 |
| lifeless | Alex_Gaynor: hi | 19:22 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add groups-staging.openstack.org node https://review.openstack.org/42608 | 19:22 |
| *** olaph has joined #openstack-infra | 19:23 | |
| *** apcruz has quit IRC | 19:27 | |
| *** fbo_away is now known as fbo | 19:27 | |
| mordred | clarkb, Alex_Gaynor: I +2'd https://review.openstack.org/#/c/42692/ - looks good - but I remember seeing a question about making it silent first in the scrollback | 19:28 |
| *** pabelanger has joined #openstack-infra | 19:30 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add job to propose tag merges https://review.openstack.org/41927 | 19:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Voting: false for py33, pylint for tuskarclient https://review.openstack.org/42632 | 19:34 |
| clarkb | ekarlso-: yes it is very slow. we throw more slaves at it which causes gerrit git to slow down... we are in the process of throwing more resources at that problem | 19:39 |
| clarkb | jeblair: lunch was a little quicker than anticipated. I am ready to switch over to git.o.o whenever you are | 19:40 |
| *** nati_uen_ has quit IRC | 19:41 | |
| *** tjones has joined #openstack-infra | 19:42 | |
| *** renier_ has joined #openstack-infra | 19:42 | |
| *** nati_ueno has joined #openstack-infra | 19:42 | |
| dansmith | dtroyer: is there something proposed for the glance breakage? | 19:43 |
| clarkb | dansmith: https://review.openstack.org/#/c/42706 | 19:43 |
| dtroyer | dansmith: yes, https://review.openstack.org/#/c/42706/. I'm trying to scare up markwash but I think he's at lunch | 19:43 |
| clarkb | dtroyer: jeblair: should we consider kicking everything out of the gate? | 19:43 |
| dansmith | is he going to approve with that all-one-line commit message? :) | 19:44 |
| clarkb | so that if 42706 is good to go it doesn't get delayed? of course keeping everything else out of the gate is probably more trouble than it is worth | 19:44 |
| dtroyer | that's how the revert-change button worked…I should try to fix it. | 19:44 |
| dansmith | ah | 19:44 |
| dansmith | then maybe it's broken in the original commit? | 19:44 |
| dtroyer | no, that's what I put in the message box | 19:44 |
| dansmith | ah | 19:45 |
| jeblair | clarkb: when it's actionable, we can kick things out. | 19:45 |
| dansmith | well, I just wanted to know if I should be reverifying things yet.. :P | 19:45 |
| * dansmith sits quietly in the corner | 19:45 | |
| *** renier_ has left #openstack-infra | 19:46 | |
| clarkb | pleia2: what was wrong with 1000 cache entries? was it too much? | 19:46 |
| jeblair | clarkb: how about i pick a precise slave, shut down puppet, and manually install the new gerrit-git-prep there to test? | 19:47 |
| pleia2 | clarkb: for me in testing, yeah it was too much, but with lots of users it may e ok | 19:47 |
| pleia2 | bs | 19:47 |
| clarkb | jeblair: sure | 19:47 |
| pleia2 | BE | 19:47 |
| pleia2 | typing! | 19:47 |
| dtroyer | damn, that created another review... | 19:47 |
| clarkb | pleia2: were you having memory issues? | 19:47 |
| pleia2 | clarkb: no, but I didn't stress test it much | 19:47 |
| pleia2 | it's on a hpcloud instance with 8G of ram | 19:48 |
| pleia2 | (my test one) | 19:48 |
| clarkb | pleia2: what exactly was wrong with it? I guess I don't understand what issues were seen with it | 19:49 |
| dtroyer | for the record, the current glanceclient revert review is https://review.openstack.org/#/c/42720/ | 19:50 |
| pleia2 | clarkb: what was wrong with caching? it just cached too much, I'd be working on things, make a change to a config, cache takes forever to turn over with one user | 19:50 |
| pleia2 | so my pages wouldn't update with config changes, changes to repos | 19:50 |
| clarkb | gotcha, so TTLs need tuning | 19:50 |
| pleia2 | yeah | 19:51 |
| pleia2 | by default on centos the only cache thing specifically defined was the cache-size, I didn't play around with the other caching options | 19:51 |
| *** sandywalsh has quit IRC | 19:53 | |
| *** derekh has joined #openstack-infra | 19:54 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Enable cgit caching. https://review.openstack.org/42723 | 19:54 |
| clarkb | pleia2: ^ that lowers the cache TTL for some of the dynamic things that can be cached | 19:54 |
| clarkb | pleia2: looking in cgitrc(5) those two values seemed like they would be the most dynamic for us | 19:55 |
| pleia2 | clarkb: wfm, want to toss in one for cache-root-ttl=1 too? (that should cache the front page too) | 19:55 |
| clarkb | pleia2: I can add that too. How often does the front page change? | 19:55 |
| pleia2 | clarkb: only when projects are added/removed | 19:56 |
| pleia2 | and changed I guess, since it has a minute by minute timestamp | 19:56 |
| pleia2 | in the Idle column | 19:56 |
| jeblair | jog0: http://paste.openstack.org/show/44542/ | 19:56 |
| jeblair | jog0: do you have any idea what that novaclient exception means? | 19:56 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Enable cgit caching. https://review.openstack.org/42723 | 19:57 |
| clarkb | pleia2: ^ that lowers the root ttl to one minute | 19:57 |
| pleia2 | clarkb: aah, I just realized it defaults to 5 | 19:59 |
| clarkb | yeah just about everything does but the scan ttl | 19:59 |
| pleia2 | clarkb: yeah, I got this mixed up with scan, sorry | 19:59 |
| clarkb | I think we can try these values as they shouldn't affect the change jeblair is testing | 19:59 |
| clarkb | pleia2: you think the scan ttl should be lower too? | 20:00 |
| pleia2 | sounds good | 20:00 |
| pleia2 | clarkb: no, I think it's good | 20:00 |
| clarkb | I am personally not too worried about what happens when new projects are added as there are enough moving parts that trying to keep everything in sync is a losing battle | 20:00 |
| *** colinmcnamara has joined #openstack-infra | 20:00 | |
| clarkb | keeping stuff within reasonable bounds is doable though | 20:00 |
| jeblair | clarkb: can you review and approve https://review.openstack.org/#/c/42691/ ? | 20:01 |
| clarkb | yes | 20:01 |
| jeblair | clarkb: (i do want your eyes on it, but it seems to work on precise1) | 20:01 |
| jeblair | clarkb: (i tested both parts; they should be able to apply independently and keep working) | 20:01 |
| clarkb | jeblair: done | 20:02 |
| clarkb | jeblair: yeah I actually reviewed it | 20:02 |
| clarkb | made sure there were no errant GERRIT_SITEs left behind and so on | 20:02 |
| jeblair | clarkb, mordred: the novaclient object for hpcloud-az1 barfed and is now permanently broken | 20:03 |
| jeblair | (in the running nodepool) | 20:03 |
| clarkb | one thing we need to be careful with is that for any of the jobs running on proposal that we don't try to push to a git:// remote | 20:03 |
| clarkb | I checked that the existing jobs don't do that and I think we are good (they use git review) | 20:03 |
| jeblair | cool | 20:04 |
| jeblair | (yet another reason for gerrit and origin to be different remotes) | 20:04 |
| clarkb | yup | 20:04 |
| *** thomasbiege has joined #openstack-infra | 20:04 | |
| clarkb | jeblair: permanently within the state of the process or even if new objects are created? | 20:04 |
| jeblair | clarkb: well, we never recreate client objects... actually... that does suggest a way to fix it | 20:04 |
| jeblair | clarkb: we could change an authentication value, then change it back | 20:05 |
| openstackgerrit | A change was merged to openstack-infra/config: Have gerrit-git-prep use git.o.o https://review.openstack.org/42691 | 20:05 |
| jeblair | clarkb: that should give us a new novaclient for az1 | 20:05 |
| *** sandywalsh has joined #openstack-infra | 20:06 | |
| jeblair | SpamapS, jog0: you may also be interested in bug 1214118 | 20:07 |
| uvirtbot | Launchpad bug 1214118 in python-novaclient "ClientException: <attribute 'message' of 'exceptions.BaseException' objects>" [Undecided,New] https://launchpad.net/bugs/1214118 | 20:08 |
| jeblair | SpamapS, jog0: which we're now seeing in our very heavy novaclient use | 20:08 |
| SpamapS | bleh | 20:08 |
| clarkb | jeblair: is that change worth puppeting or do you just want to ninja it in? | 20:09 |
| jeblair | clarkb: i think it's more of a cowboy change | 20:09 |
| clarkb | jeblair: that hack seems sound to me. We will leak the file descriptors but there aren't too many of them right? | 20:10 |
| jeblair | clarkb: yep | 20:11 |
| jeblair | clarkb: it currently has 100 fds | 20:11 |
| morganfainberg | clarkb: i'm not sure if this should be brought up with infra or glance or devstack: bug 1214094 is causing a ton of failures in granade gate checks (--public parameter was deprecated in glance), at least that appears to be the issue. | 20:15 |
| uvirtbot | Launchpad bug 1214094 in devstack "glance --public parameter has been deprecated" [Undecided,New] https://launchpad.net/bugs/1214094 | 20:15 |
| morganfainberg | gate/jenkins etc. | 20:15 |
| clarkb | morganfainberg: dtroyer has proposed a fix | 20:15 |
| morganfainberg | clarkb: awesome. | 20:16 |
| clarkb | morganfainberg: https://review.openstack.org/#/c/42706 | 20:16 |
| morganfainberg | clarkb: thanks | 20:16 |
| clarkb | morganfainberg: I think the gate will basically be spinning wheels until that change or something else addressing the problem merges | 20:16 |
| morganfainberg | clarkb: yep. good to know. | 20:16 |
| morganfainberg | clarkb: at least someone is aware of it / working on it. | 20:17 |
| morganfainberg | i just didn't want it to lay around with no one aware. | 20:18 |
| clarkb | jeblair: did you update the precise27 slave's DNS records? | 20:18 |
| clarkb | morganfainberg: yup, no problem | 20:18 |
| jeblair | clarkb: negative | 20:18 |
| * clarkb tries to remember how to use statusbot | 20:19 | |
| clarkb | jeblair: is it #status notice somemessage here? | 20:19 |
| *** xBsd has quit IRC | 20:19 | |
| clarkb | jeblair: I can add the DNS records if you like | 20:19 |
| jeblair | clarkb: if you want to broadcast | 20:19 |
| clarkb | jeblair: I figure broadcasting that hte gate is sideways while grenade gets sorted may be helpful | 20:20 |
| jeblair | clarkb: that does not set the topic | 20:20 |
| jeblair | clarkb: alert sets topics | 20:20 |
| clarkb | ah, we should probably set the topic too. | 20:20 |
| clarkb | I will use alert | 20:20 |
| jeblair | clarkb: dns updated | 20:20 |
| clarkb | #status alert gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix: https://review.openstack.org/#/c/42720/ | 20:22 |
| clarkb | status bot has left us again :) | 20:22 |
| clarkb | er :( | 20:22 |
| *** sandywalsh has quit IRC | 20:23 | |
| *** vipul is now known as vipul-away | 20:23 | |
| *** vipul-away is now known as vipul | 20:23 | |
| morganfainberg | poor status bot. | 20:24 |
| morganfainberg | it just wants to feel loved. | 20:24 |
| clarkb | jeblair: I am going to try starting statusbot with its initscript | 20:25 |
| jeblair | clarkb: hrm, the recreate-client trick didn't work. | 20:25 |
| clarkb | jeblair: could it be an issue with the account? | 20:26 |
| jeblair | clarkb: (i mean, it logged messages saying it recreated it, but the (presumably new) client is still throwing the same errors) | 20:26 |
| clarkb | mordred: you around? | 20:26 |
| jeblair | clarkb: i will try nova deleting something | 20:26 |
| *** openstackstatus has joined #openstack-infra | 20:27 | |
| clarkb | hurray statusbot has returned | 20:27 |
| clarkb | #status alert gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix: https://review.openstack.org/#/c/42720/ | 20:27 |
| mrmartin | jeblair: it was accepted by Jenkins after puppet-lint cleanup. https://review.openstack.org/#/c/42608/ | 20:27 |
| openstackstatus | NOTICE: gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix: https://review.openstack.org/#/c/42720/ | 20:27 |
| *** ChanServ changes topic to "gate-grenade-devstack-vm is currently failing preventing merges. Proposed fix: https://review.openstack.org/#/c/42720/" | 20:27 | |
| jeblair | clarkb: the nova delete _is_ taking a long time and has not returned | 20:28 |
| clarkb | I wonder if we long into the gui if it will show us a big red banner | 20:28 |
| * clarkb tries | 20:28 | |
| jeblair | clarkb: az1 only if i didn't mention | 20:29 |
| jeblair | # nova delete 2151767 | 20:29 |
| jeblair | ERROR: n/a (HTTP 500) | 20:29 |
| jeblair | i think i understand the novaclient exception now. | 20:29 |
| jeblair | there is no message. | 20:29 |
| clarkb | interesting | 20:30 |
| clarkb | gui is taking forever to open az1 compute | 20:30 |
| clarkb | jeblair: that errored too. I think this problem may be independnet of node pool | 20:31 |
| *** harlowja_ has quit IRC | 20:31 | |
| jeblair | clarkb: this is where if someone from hpcloud were in here, we could let them know their cloud was broken. | 20:31 |
| clarkb | jeblair: ya | 20:31 |
| *** harlowja has joined #openstack-infra | 20:31 | |
| clarkb | jeblair: as is I will try walking down the hall to see if anyone can help | 20:31 |
| clarkb | brb | 20:31 |
| *** w_ has joined #openstack-infra | 20:31 | |
| jeblair | clarkb: the zuul graph says we launched apprx 600 jobs last hour. i believe that's the first time we've topped 500? | 20:33 |
| *** olaph has quit IRC | 20:33 | |
| clarkb | jeblair: no, I think we have gone over at least one other time | 20:35 |
| clarkb | but it doesn't happen often | 20:35 |
| *** sandywalsh has joined #openstack-infra | 20:36 | |
| *** vipul is now known as vipul-away | 20:38 | |
| *** w_ has quit IRC | 20:39 | |
| clarkb | jeblair: load on review.o.o seems to be falling. Have you been keeping an eye on the application of the g-g-p update? | 20:40 |
| jeblair | clarkb: just started checking in on it | 20:40 |
| *** w_ has joined #openstack-infra | 20:42 | |
| clarkb | jeblair: once you are happy with it I would like to try merging the cgit cache change https://review.openstack.org/#/c/42723/ | 20:43 |
| jeblair | clarkb: jenkins-jobs are running on 01 and 02 | 20:43 |
| jeblair | clarkb: i'm sure i will need to re-run them | 20:43 |
| jeblair | because of the timeout | 20:43 |
| jeblair | clarkb: i'm also checking to make sure the script is updated on all the slaves | 20:44 |
| clarkb | jeblair: cool. should I start checking centos and you precise or odds/evens? | 20:44 |
| jeblair | clarkb: would you mind checking the centos and py3k? | 20:44 |
| jeblair | clarkb: i'll finish up precise | 20:44 |
| clarkb | will do | 20:44 |
| *** dkliban has quit IRC | 20:45 | |
| jeblair | clarkb: md5sum 5bbb93f4559b947bf1959591a636f12c /usr/local/jenkins/slave_scripts/gerrit-git-prep.sh | 20:45 |
| *** vipul-away is now known as vipul | 20:46 | |
| *** w_ has quit IRC | 20:47 | |
| mordred | clarkb: sup | 20:48 |
| jeblair | clarkb: precise nodes check out; i'm running jenkins jobs on 01 and 02 now | 20:49 |
| clarkb | jeblair: centos nodes are fine | 20:49 |
| clarkb | precisepy3k-4 is the only one that I found with a problem. Correcting it now | 20:50 |
| *** w_ has joined #openstack-infra | 20:50 | |
| jeblair | clarkb: wow, jenkins-jobs claimed it had nothing to do. is it possible that job update run actually completed? | 20:50 |
| clarkb | mordred: HP AZ1 went sideways. I hopped on hipchat and am sort of following along. tl;dr we need to ride it out | 20:51 |
| jeblair | clarkb, pleia2: reviewing cache change now | 20:51 |
| clarkb | mordred: I figured you might have some great insight | 20:51 |
| mordred | clarkb: ah. ok. so, I just started reading scrollback - but the underlying cause is az1 itself going batty? | 20:51 |
| clarkb | mordred: yup | 20:52 |
| *** thomasbiege has quit IRC | 20:53 | |
| *** thomasbiege has joined #openstack-infra | 20:53 | |
| jeblair | clarkb pleia2: change lgtm; what about setting cache-static-ttl to something high, like 15? | 20:53 |
| clarkb | jeblair: py3k-4 is good now | 20:53 |
| clarkb | jeblair: we can do that. | 20:53 |
| clarkb | jeblair: my only concern was using up all of the available cache entries and not cleaning them up quickly | 20:54 |
| clarkb | jeblair: but we can increase 1000 to something else if we notice that happening | 20:54 |
| jeblair | clarkb: ok, let's go with it as-is and see what it looks like | 20:54 |
| mordred | ok. I think I'm caught p | 20:54 |
| mordred | up | 20:54 |
| jeblair | clarkb: aprvd | 20:54 |
| pleia2 | I figure we'll modify the cache stuff after some testing | 20:54 |
| clarkb | jeblair: thanks | 20:54 |
| pleia2 | thanks jeblair | 20:54 |
| jeblair | pleia2: some what now? :) | 20:55 |
| pleia2 | hehe | 20:55 |
| pleia2 | live testing! | 20:55 |
| jeblair | pleia2: yes, we're about to throw a bunch of load at it much sooner than i would have expected. :) | 20:55 |
| pleia2 | go go git | 20:55 |
| jeblair | in fact. the load average on git.o.o is currently 37. | 20:55 |
| clarkb | INCOMING! | 20:55 |
| * mordred wonders if jeblair has been replaced by mordred | 20:55 | |
| pleia2 | oh boy | 20:55 |
| clarkb | jeblair: it is 66 on review.o.o | 20:56 |
| mordred | oh wow. 37 on git.o.o? | 20:56 |
| jeblair | clarkb: yay we distributed load! | 20:56 |
| *** w_ has quit IRC | 20:56 | |
| *** olaph has joined #openstack-infra | 20:56 | |
| pleia2 | mordred: anteaya found this for you and jeblair http://imgur.com/SI6h3U9 | 20:57 |
| *** thomasbiege has quit IRC | 20:57 | |
| *** thomasbiege has joined #openstack-infra | 20:57 | |
| clarkb | jeblair: https://review.openstack.org/#/c/42720/ was approved, but with the current state of the gate it will be hours before merging | 20:57 |
| clarkb | jeblair: I think now might be a good time to kill zuul and start this train over | 20:58 |
| jeblair | clarkb: actually, i think it will never merge | 20:59 |
| jeblair | clarkb: i think we need to bypass testing on that one | 20:59 |
| jeblair | clarkb: https://jenkins01.openstack.org/job/gate-grenade-devstack-vm/2458/console | 20:59 |
| mordred | jeblair: does it need to merge before it can merge? | 20:59 |
| *** colinmcnamara has quit IRC | 21:00 | |
| clarkb | jeblair: fun | 21:00 |
| jeblair | mordred: yep. | 21:00 |
| mordred | and do we need to accelerate backporting the requirements updates to stable/grizzly? | 21:00 |
| jeblair | clarkb, mordred: if you are in agreement, i'll make it so | 21:00 |
| mordred | jeblair: ok. I'm second the vote to force merge it | 21:00 |
| clarkb | jeblair: yes, it appears that the upgrade testing has us in an unhappy place unti lthat goes in | 21:00 |
| jeblair | mordred: that's not what caused this to get by | 21:01 |
| *** emagana has joined #openstack-infra | 21:01 | |
| clarkb | mordred: iirc it has to do with how grenade works | 21:01 |
| clarkb | grenade is using merged code for the clients | 21:01 |
| jeblair | mordred: the combo that is not tested is "new client lib being used by stable devstack branch" | 21:01 |
| clarkb | and not testing proposed code | 21:01 |
| jeblair | clarkb: that's not correct | 21:02 |
| clarkb | oh | 21:02 |
| jeblair | clarkb: it's half-correct | 21:02 |
| jeblair | clarkb: it is using master for the setup, and proposed code for the new run | 21:02 |
| clarkb | but since master is broken we still have half a broken setup | 21:03 |
| jeblair | clarkb: master glanceclient doesn't work with stable openstack | 21:03 |
| jeblair | devstack in this case | 21:04 |
| mordred | jeblair: ah | 21:04 |
| *** dkliban has joined #openstack-infra | 21:04 | |
| mordred | jeblair: so, in theory, if we get the "gate trunk client vs. stable servers" in it should trap this | 21:04 |
| jeblair | anyway, it could be fixed by using the proposed code for the clients in the grenade setup, or via a 'test proposed client changes against stable branches of servers (and devstack" job. | 21:04 |
| jeblair | mordred: yep | 21:05 |
| mordred | jeblair: might be a good test case, btw... | 21:05 |
| jeblair | yeah | 21:05 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable cgit caching. https://review.openstack.org/42723 | 21:06 |
| jeblair | https://review.openstack.org/#/c/42720/ is merged | 21:06 |
| *** derekh has quit IRC | 21:07 | |
| clarkb | cool. once the gate resets itself we should see things start flowing again. We can short circuit that be killing a test running in the change at the top of the queue. I will go ahead and do that I think | 21:07 |
| jeblair | clarkb: the one at the top may merge | 21:07 |
| clarkb | actually. I will wait since the head of the queue is neutron which can merge | 21:07 |
| clarkb | jinx | 21:07 |
| clarkb | I will do it to the second change | 21:07 |
| jeblair | clarkb: sounds good | 21:07 |
| *** woodspa has quit IRC | 21:08 | |
| *** colinmcnamara has joined #openstack-infra | 21:08 | |
| clarkb | all done. | 21:08 |
| *** olaph has quit IRC | 21:08 | |
| clarkb | sorry sacrificial nova change. It was for the greater good | 21:09 |
| *** olaph has joined #openstack-infra | 21:09 | |
| *** mrmartin has quit IRC | 21:10 | |
| * mordred bows his head in sorry for the nova change | 21:11 | |
| mordred | or in sorrow | 21:11 |
| mordred | dammit | 21:11 |
| * mordred blames the dubstep and the boxes covered in playa dust filling his livingroom | 21:11 | |
| *** SergeyLukjanov has quit IRC | 21:11 | |
| *** thomasbiege has quit IRC | 21:12 | |
| *** pabelanger has quit IRC | 21:13 | |
| *** w_ has joined #openstack-infra | 21:13 | |
| *** pentameter has quit IRC | 21:13 | |
| Alex_Gaynor | clarkb: so it looks like I got the ppa syntax wrong in teh puppet file, is there an example of this or some docs I should look at? | 21:14 |
| *** olaph has quit IRC | 21:14 | |
| clarkb | Alex_Gaynor: there is an example. let me find it | 21:14 |
| Alex_Gaynor | thanks | 21:15 |
| mordred | Alex_Gaynor: modules/salt/manifests/init.pp | 21:16 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Add support for running tox with PyPy and use it for Marconi https://review.openstack.org/42692 | 21:16 |
| Alex_Gaynor | mordred: sigh, was missing a colon | 21:16 |
| clarkb | load on review.o.o is ~171 | 21:16 |
| clarkb | poor gerrit | 21:17 |
| Alex_Gaynor | clarkb: gerrit is a computer, i"m sure it doens't mind | 21:17 |
| mordred | clarkb: load on git.o.o is 0.25 | 21:17 |
| mordred | Alex_Gaynor: it's not a computer - it's a virtual computer | 21:17 |
| Alex_Gaynor | mordred: sometimes my friends ask me what rackspace does, my answer is "You know how some companies will sell you a computer? We'll rent you a small fraction of a computer." | 21:17 |
| mordred | Alex_Gaynor: nice | 21:18 |
| mordred | I have the hardest time describing what I do to my friends who mostly work in theatre | 21:18 |
| Alex_Gaynor | (not taolly true since we have the dedicated business, but uslally good for a laugh) | 21:18 |
| *** boris-42 has quit IRC | 21:18 | |
| clarkb | looks like it is still git load. jeblair do we need to update zuul to talk to git.o.o as well?a | 21:18 |
| jeblair | clarkb: no, it's the devstack jobs | 21:18 |
| mordred | I don't run a cloud, I don't really even develop the software that runs a cloud - I develop software that helps people develop software that people can use to run computers that let people run computers to run their applications | 21:19 |
| clarkb | oh yes, the other significant portion of our load | 21:19 |
| clarkb | jeblair: did the d-g change merge? | 21:19 |
| jeblair | clarkb: it hasn't been approved | 21:19 |
| jeblair | it just finished it's second round of tests | 21:19 |
| jeblair | https://review.openstack.org/#/c/42693/ | 21:19 |
| mordred | and failed grenade | 21:19 |
| jeblair | as expected | 21:19 |
| clarkb | jeblair: I think we should force merge that one too if we want it in quick | 21:19 |
| clarkb | otherwise it may get stuck in teh gate (which isn't the end of the world now that grenade should be happy) | 21:20 |
| jeblair | clarkb: i don't think we're that desperate | 21:20 |
| clarkb | I am going to approve it | 21:20 |
| jeblair | clarkb: ok. the output looks correct | 21:20 |
| mordred | jeblair: did I miss something in the scrollback about why git usage against gerrit suddenly just went apeshit today? did we just hit a crazy usage spike coupled with the grenade failures? | 21:21 |
| clarkb | mordred: thats it | 21:21 |
| jeblair | mordred: we got very good at running lots of jobs; i don't think grenade was a factor | 21:21 |
| clarkb | mordred: the 21st is the last day to propose feature change sto nova and cinder. so everyone is pushing nova changes | 21:21 |
| clarkb | mordred: if you look at the check queue it is like 95% nova | 21:22 |
| jeblair | mordred: i think it's mostly nodepool letting us scale up to the point where we can meet the load for the feature freeze | 21:22 |
| *** rfolco has quit IRC | 21:22 | |
| jeblair | (which was why i have been rushing to get nodepool into place) | 21:22 |
| *** w_ has quit IRC | 21:22 | |
| jeblair | and of course we've found our next scalability bottelneck. :) | 21:22 |
| Alex_Gaynor | mordred: FTR one of the pypy developer's reactions to your job description: "I develop software for people like that" | 21:23 |
| clarkb | pleia2: does apache need to be restarted to pick up the cgit changes? | 21:23 |
| pleia2 | clarkb: not in cgitrc | 21:23 |
| mordred | jeblair: nice! | 21:23 |
| mordred | Alex_Gaynor: haha. :) | 21:23 |
| *** dina_belova has quit IRC | 21:23 | |
| clarkb | pleia2: I see it now. I had to access some pages to create cache items | 21:24 |
| clarkb | pleia2: I guess I had let them all time out | 21:24 |
| jeblair | clarkb: earlier i straced a git upload-pcak process. it was very slowly reading all of the refs directories | 21:24 |
| * pleia2 nods | 21:24 | |
| jeblair | there are about 9000 in nova | 21:24 |
| jeblair | i wonder if we should start packing refs? | 21:24 |
| clarkb | jeblair: doesn't the gc do that for us? | 21:25 |
| jeblair | i'm slightly concerned about that because i believe they are very hard (eg, you can not do it with normal git commands) to unpack. | 21:25 |
| mordred | jeblair: what does packing refs mean/do? | 21:25 |
| jeblair | the only reason we were able to remove the zuul refs is because they aren't being packed | 21:25 |
| clarkb | mordred: convert the loose object files into pack files which are compressed and contain a bunch of loose files | 21:25 |
| jeblair | mordred: flattens the refs filesystem hieararchy into files with lists of refs | 21:25 |
| jeblair | clarkb: not objects, _refs_ | 21:26 |
| jeblair | clarkb: we are packing objects | 21:26 |
| clarkb | ooooohhhhhhhh | 21:26 |
| jeblair | https://www.kernel.org/pub/software/scm/git/docs/git-pack-refs.html | 21:26 |
| clarkb | looks like only branch heads will automagically unpack (which makes sense as they move) | 21:27 |
| * mordred wonders what this might do to the refs/for namespace and/or is there a way to filter that from our replication? | 21:27 | |
| jeblair | i'm also not sure what would happen if we moved a tag. | 21:28 |
| jeblair | mordred: refs/for is not a real namespace | 21:28 |
| jeblair | mordred: are you thinking of changes? | 21:28 |
| mordred | yes | 21:28 |
| clarkb | gerrit MiTMs refs/for | 21:28 |
| jeblair | that can be filtered. i'm not sure we want to filter it though. | 21:28 |
| clarkb | is gerrit's FS ext3? | 21:29 |
| jeblair | i like the idea that the full change history is available in the git repo. | 21:29 |
| clarkb | maybe we can make stating lots of files less painful by changing the file system? | 21:29 |
| mordred | if cgit is taking a lot of time to walk refs trees - it could be on account of the massive number of refs/changes - it might be interesting to see what the difference in serving a repo with and without them is | 21:29 |
| mordred | jeblair: I agree - I'm not sure we want to filter it either | 21:29 |
| mordred | but I'm curious as to what the cost of replicating it everywhere is | 21:30 |
| Alex_Gaynor | Is there anyone who knows stuff about testr besides lifeless ? | 21:30 |
| jeblair | mordred: i would expect that if they are not there, it would not take long to look at them. | 21:30 |
| mordred | Alex_Gaynor: how deep do you need to know? | 21:30 |
| clarkb | Alex_Gaynor: I end up helping out as much as I can but I definitely don't know as much as lifeless | 21:30 |
| Alex_Gaynor | mordred: I need to know why under pypy it prints: 'memoryview' object has no attribute 'decode' | 21:30 |
| jeblair | mordred, clarkb: i'd like to understand the problem more. git.o.o is so unloaded now, it's hard to dig deeper | 21:31 |
| clarkb | Alex_Gaynor: does memoryview need a tobytes() first? | 21:32 |
| * clarkb greps for memoryview in testr | 21:32 | |
| Alex_Gaynor | clarkb: no clue! I can't imagine what pypy is doing differently than CPython here | 21:32 |
| mordred | Alex_Gaynor, clarkb: subunit uses memoryview | 21:32 |
| lifeless | Alex_Gaynor: subunit/v2.py | 21:33 |
| lifeless | Alex_Gaynor: if safe_hasattr(__builtins__, 'memoryview'): | 21:33 |
| lifeless | Alex_Gaynor: body = memoryview(packet[-1]) | 21:33 |
| mordred | Alex_Gaynor: and around line 456 | 21:33 |
| mordred | return utf8_bytes.decode('utf-8'), length+pos | 21:33 |
| mordred | utf8_bytes is a slide of that body variable lifeless pasted | 21:34 |
| mordred | slice | 21:34 |
| Alex_Gaynor | lifeless: now that you're here! What should I comment out to get the full TB instead of just printing the error? | 21:34 |
| lifeless | Alex_Gaynor: I ponged you a couple hours back :) | 21:34 |
| Alex_Gaynor | lifeless: doh :( sorry I missed it | 21:34 |
| mordred | Alex_Gaynor: my favorite - traceback hiding :) | 21:34 |
| jeblair | clarkb: did you add precise27? | 21:34 |
| clarkb | jeblair: no sorry. I will do that now | 21:35 |
| jeblair | clarkb: no problem; it looks like dns is updated | 21:35 |
| clarkb | jeblair: actually it seems to have added itself | 21:35 |
| jeblair | ah of course. :) | 21:35 |
| lifeless | Alex_Gaynor: export TESTR_PDB=1 | 21:35 |
| clarkb | since I never removed the node def in jenkins | 21:35 |
| clarkb | A memoryview supports slicing to expose its data. Taking a single index will return a single element as a str object. Full slicing will result in a subview: I wonder if pypy returns a subview each time | 21:36 |
| clarkb | lifeless: Alex_Gaynor ^ | 21:36 |
| *** w_ has joined #openstack-infra | 21:36 | |
| mordred | lifeless: how do I run python-subunit's tests? | 21:36 |
| *** dkranz has quit IRC | 21:36 | |
| clarkb | make | 21:36 |
| Alex_Gaynor | clarkb: nope, that seems to work ok | 21:36 |
| jeblair | clarkb, mordred: perhaps when the gate resets in a minute or so, git.o.o will get slammed and we can see it slow down again | 21:37 |
| lifeless | Alex_Gaynor: whats the bigger context here? | 21:37 |
| mordred | jeblair: that would be nice | 21:37 |
| clarkb | jeblair: it won't get slammed by d-g jobs though, just unittests and docs and things | 21:37 |
| mordred | lifeless: Alex_Gaynor is trying to run marconi's tests using pypy | 21:37 |
| Alex_Gaynor | lifeless: running setup.py testr under pypy doesn't work | 21:37 |
| lifeless | Alex_Gaynor: like, why is testr running under pypy? (It should work, and I'll happily take bugs about it not working) | 21:37 |
| Alex_Gaynor | mordred: actually marconi works, this is novaclient, marconi isn't on testr | 21:37 |
| lifeless | Alex_Gaynor: but you don't need to run testr under pypy to run marconi's tests under testr. | 21:37 |
| mordred | Alex_Gaynor: ah. neat | 21:37 |
| mordred | lifeless: tox | 21:37 |
| Alex_Gaynor | lifeless: well, it all uses the python from tox, so it's just easier to have it be the same python | 21:38 |
| mordred | lifeless: is how we switch python impls | 21:38 |
| lifeless | Alex_Gaynor: testr is a meta-runner, the test processes are a separate process. | 21:38 |
| lifeless | ah | 21:38 |
| jeblair | clarkb: everything behind the devstack-gate change will use it. :) | 21:38 |
| mordred | lifeless: also, on most of our boxes, testr won't exist outside of the virtualenv, which needs to be created with the python that is going to run the tests | 21:38 |
| jeblair | which is nothing atm. | 21:39 |
| annegentle | did we switch over where the maven plugin is loaded from? I'm seeing [INFO] Unable to find resource 'org.apache.maven.plugins:maven-clean-plugin:pom:2.3' in repository repo.jenkins-ci.org ( | 21:39 |
| annegentle | http://logs.openstack.org/77/41777/6/check/gate-openstack-manuals-validate/384352c/console.html | 21:39 |
| Alex_Gaynor | lifeless: I found the problem, this is making me rage a bit, where's the testr repo, I'll send you a patch which should fix it (tl;dr; the only reason it works under cpython is that your `hashattr(__builtins__, 'memoryview')` check is a bit broken) | 21:41 |
| clarkb | annegentle: no, your job should still be using the Maven3 build env in Jenkins and unless you changed your project or one of your upstreams changed will go to the same place | 21:41 |
| lifeless | Alex_Gaynor: launchpad.net/testrepository | 21:41 |
| clarkb | annegentle: I am not sure what is going on there yet | 21:41 |
| Alex_Gaynor | lifeless: uh oh, do I have to learn how to use bzr? | 21:41 |
| annegentle | clarkb: yea I'm digging deeper too | 21:41 |
| lifeless | Alex_Gaynor: no, you can just file a bug with a patch | 21:41 |
| Alex_Gaynor | lifeless: can I ask the really dumb question "how do I check this out" | 21:42 |
| mordred | Alex_Gaynor: "bzr branch lp:testrepository ; cd testrepository ; <fix> ; bzr commit ; bzr push lp:~alexgaynor/testrepository/fix-the-bug" | 21:42 |
| lifeless | Alex_Gaynor: bzr branch lp:testrepository - but the bug isn't in testrepository. | 21:42 |
| lifeless | Alex_Gaynor: it's in subunit | 21:42 |
| Alex_Gaynor | lifeless: err, right, sorry | 21:42 |
| Alex_Gaynor | lifeless: where's the subunit repo :) | 21:42 |
| mordred | Alex_Gaynor: "bzr branch lp:subunit ; cd subunit ; <fix> ; bzr commit ; bzr push lp:~alexgaynor/subunit/fix-the-bug" | 21:43 |
| lifeless | Alex_Gaynor: so, I need to cast bug to a str before decoding? | 21:43 |
| Alex_Gaynor | lifeless: yup, I'll also fix your hasattr check :) the only reason this doesn't break on cpython is that hasattr check never retuirns true | 21:43 |
| Alex_Gaynor | lifeless: __builtins__ is sometimes a dict on cpython and sometimes a module, here it happens to be a dict, so of course hasaatrr doesn't work | 21:43 |
| lifeless | Alex_Gaynor: oh, interactive vs in a module? | 21:44 |
| lifeless | Alex_Gaynor: don't bother with a patch, I've got the changes made here | 21:44 |
| marun | lifeless: ping | 21:44 |
| mordred | Alex_Gaynor: (actually, fwiw) "bzr branch lp:subunit ; cd subunit ; <fix> ; bzr commit ; bzr push lp:~king_kilr/subunit/fix-the-bug" | 21:44 |
| Alex_Gaynor | lifeless: something like that, you want `import __bujiltin__` hasattr(__builtin__, 'memoryview') | 21:44 |
| Alex_Gaynor | lifeless: except on python3 the module is named builtin (no dunder) | 21:44 |
| jeblair | clarkb, mordred: i can reproduce the slowness by cloning nova 10 times | 21:45 |
| mordred | jeblair: nice! | 21:45 |
| mordred | from git.o.o ? | 21:45 |
| mordred | that's excellent news | 21:45 |
| lifeless | Alex_Gaynor: I have a few reviews to finish, but I'll push up a subunit release today for you | 21:45 |
| *** pblaho has quit IRC | 21:45 | |
| Alex_Gaynor | lifeless: thanks, you rock! (let me know if you want me to do a once over a on patch) | 21:45 |
| lifeless | Alex_Gaynor: I presume on pypy __builtins__ is always an object ? | 21:45 |
| Alex_Gaynor | lifeless: yup | 21:45 |
| lifeless | 'yay'. | 21:45 |
| Alex_Gaynor | lifeless: http://docs.python.org/2/library/__builtin__.html#module-__builtin__ see the note there :) | 21:46 |
| Alex_Gaynor | lifeless: __builtin__ (no s) always works | 21:46 |
| jeblair | ecxept i'm getting a bit of packet loss | 21:46 |
| jeblair | which makes it difficult | 21:46 |
| mordred | ew | 21:46 |
| jeblair | strace -p 23445 | 21:48 |
| jeblair | mordred: ^ | 21:48 |
| marun | lifeless: is parameterization of a fixture created with testresources (e.g. VmFixture(tenant_id)) a good idea? Is reuse possible? | 21:48 |
| mordred | jeblair: on which server? | 21:49 |
| mordred | jeblair: [mordred@git ~]$ sudo strace -p 23445 | 21:49 |
| mordred | attach: ptrace(PTRACE_ATTACH, ...): No such process | 21:49 |
| jeblair | too slow | 21:49 |
| *** colinmcnamara has left #openstack-infra | 21:49 | |
| fungi | wow. had to hit pgup 58 times to get to the start of the scrollback. you're a chatty bunch | 21:49 |
| *** dprince has quit IRC | 21:49 | |
| mordred | fungi: STOP TAKING VACATION WHEN THINGS GO CRAZY ;) | 21:49 |
| jeblair | mordred: sudo screen | 21:49 |
| jeblair | mordred: sudo screen -x | 21:50 |
| clarkb | mordred: you and fungi both :P | 21:50 |
| mordred | I'm in | 21:50 |
| clarkb | me too | 21:50 |
| clarkb | jeblair: what processes are you stracing? are they upload packs? | 21:51 |
| jeblair | yes | 21:51 |
| *** xBsd has joined #openstack-infra | 21:51 | |
| mordred | uhm. so compressing the objects over and over doesn't seem like the best choice? | 21:51 |
| clarkb | mordred: well it is just once compression I think that reorts updates | 21:52 |
| clarkb | jeblair: you might want to -f that strace | 21:52 |
| fungi | btw, you know you're in seattle when you encounter a wayward mark atwood on a pike st sidewalk | 21:53 |
| clarkb | fungi: the fog into sunny weather didn't give it away earlier? | 21:53 |
| *** rfolco has joined #openstack-infra | 21:53 | |
| clarkb | jeblair: https://jenkins01.openstack.org/job/gate-oslo.config-python27/3/console | 21:54 |
| Alex_Gaynor | lifeless: untested, but http://bpaste.net/show/124350/ | 21:54 |
| Alex_Gaynor | lifeless: (if you let me know how to run the tests I can do that :P) | 21:54 |
| *** mrodden has quit IRC | 21:54 | |
| *** lbragstad has quit IRC | 21:55 | |
| *** dims has quit IRC | 21:56 | |
| jeblair | clarkb: great | 21:56 |
| clarkb | woo git | 21:57 |
| jeblair | clarkb, pleia2: any idea where relevant logs for that are? | 21:57 |
| clarkb | jeblair: that would be git daemon /me looks | 21:57 |
| lifeless | Alex_Gaynor: configure; make check | 21:57 |
| jeblair | clarkb: we use xinetd, right? | 21:57 |
| clarkb | jeblair: we do | 21:57 |
| * fungi is reviewing 38177 before he catches up on the rest of scrollback, since it could probably stand to get crammed through asap | 21:57 | |
| clarkb | jeblair: syslog | 21:58 |
| *** ftcjeff has quit IRC | 21:58 | |
| marun | who can help me with testresources? | 21:58 |
| Alex_Gaynor | lifeless: this yak stack sucks, now I have to figure out why configure doesn't work : | 21:59 |
| Alex_Gaynor | :/ | 21:59 |
| *** rfolco has quit IRC | 22:00 | |
| lifeless | Alex_Gaynor: hey, you're the one digging:) I mean, I appreciate it but since I need to do the release anyhow, I'll be doing full test runs etc | 22:01 |
| lifeless | marun: I can, mordred can, I think clarkb can. | 22:01 |
| Alex_Gaynor | lifeless: not your fault I'm deep down, but the levels of abstraction from the original problem is getting a bit too high for my comfort :) | 22:01 |
| jeblair | clarkb: do you think we should switch back off of git.o.o? or switch to https instead of git:// + | 22:01 |
| jeblair | ? | 22:01 |
| clarkb | jeblair: git-daemon[24478]: error: git-upload-pack died of signal 13 a lot of that in /var/log/messages | 22:01 |
| clarkb | jeblair: worth a shot of switching to https since the mechanism is different | 22:02 |
| clarkb | jeblair: and in theory is very similar to what we have on review.o.o. The major difference being the version of git aiui | 22:02 |
| jeblair | clarkb: aren't we running an old version of git here? | 22:03 |
| clarkb | jeblair: yes, it is slightly older | 22:03 |
| marun | lifeless: is it possible to parameterize fixture creation (e.g. NeutronVM1(tenant_id)) and have the parameter be a factor in reuse? | 22:03 |
| clarkb | jeblair: but still 1.7.1 | 22:03 |
| pleia2 | yeah, the default from centos is a bit older | 22:03 |
| clarkb | jeblair: which is new enough for smart http | 22:04 |
| *** dmakogon__ has quit IRC | 22:04 | |
| marun | lifeless: I'm not clear on how the parameter could/would be taken into account by testresources | 22:04 |
| clarkb | jeblair: I believe smart http made it into git somewhere in 1.6.X | 22:05 |
| jeblair | clarkb: oh, the signal 13's are probably me | 22:06 |
| jeblair | aborting git clones | 22:06 |
| lifeless | marun: that looks like you want a dependency resource representing the tenant | 22:06 |
| marun | lifeless: ah, so it would have to be defined at the class level | 22:06 |
| marun | lifeless: that's fine. i was figuring on having to do that for other resources anyway. | 22:07 |
| clarkb | jeblair: nice :) | 22:07 |
| clarkb | jeblair: maybe we should let git:// sitfor a little while longer? | 22:07 |
| *** burt has quit IRC | 22:09 | |
| *** qs202 has joined #openstack-infra | 22:09 | |
| clarkb | jeblair: python26 jobs for nova are spending a minute and 20 seconds in g-g-p which is much quicker than before | 22:10 |
| clarkb | so keeping git:// if possible is a good thing | 22:11 |
| jeblair | clarkb: http://paste.openstack.org/show/44545/ | 22:11 |
| jeblair | clarkb: i'm really confused | 22:11 |
| *** dims has joined #openstack-infra | 22:11 | |
| jeblair | clarkb: those are the log entries from precise23 around the time of that failed test | 22:11 |
| BobBall_Away | mordred: ping | 22:11 |
| clarkb | jeblair: weird that it is all one host | 22:12 |
| jeblair | clarkb: yeah, and none of those are for oslo.config | 22:12 |
| mordred | BobBall_Away: sup? | 22:12 |
| clarkb | jeblair: network trouble maybe? if git on the precise23 side couldn't tcp handshake with the git daemon things would derp? | 22:12 |
| mordred | clarkb: I was just about to say that | 22:12 |
| jeblair | clarkb: okay, but why did it try to update a bunch of other repos? | 22:13 |
| BobBall_Away | mordred: VIRTUAL_ENV patch - was going to upload a new version - can be multiple site-packages (one deeply nested) - will $VIRTUAL_ENV/lib/python*/site-packages always work? | 22:13 |
| clarkb | jeblair: perhaps that slave started grabbing jobs really quickly as things failed? /me looks at build history | 22:13 |
| BobBall_Away | mordred: this is the cause of the current failure in devstack because the horizon sed fails due to a newline caused by multiple site-packages | 22:14 |
| clarkb | https://jenkins01.openstack.org/computer/precise23/builds doesn't look like that is the case | 22:14 |
| mordred | BobBall_Away: uhm - wow. really there are multiple site packages? | 22:14 |
| BobBall_Away | sure - but not intentionally... | 22:14 |
| BobBall_Away | a side effect of a dodgy package ;) | 22:14 |
| mordred | ahhhhh | 22:14 |
| BobBall_Away | and a test suite ;) | 22:14 |
| clarkb | jeblair: actually it ran a glance client test, then oslo.config then neutron | 22:14 |
| mordred | BobBall_Away: what is it we're doing wiht the dir? it was a chown or something, right? | 22:15 |
| jeblair | clarkb: ok, so that neutron error seems to be from the test after the one we're interested in. | 22:15 |
| * mordred goes to look at patch | 22:15 | |
| clarkb | then glanceclient then cinder | 22:15 |
| BobBall_Away | /opt/stack/new/.venv/lib/python2.7/site-packages/flask/testsuite/test_apps/lib/python2.5/site-packages | 22:15 |
| clarkb | jeblair: yes | 22:15 |
| *** qs202 has quit IRC | 22:15 | |
| jeblair | clarkb: the previous one was glance | 22:15 |
| jeblair | clarkb: (in syslog) | 22:15 |
| BobBall_Away | mordred: to allow horizon to run in a venv we need to set HORIZON_VENV_PATH | 22:15 |
| mordred | BobBall_Away: AH - I remember this now | 22:15 |
| jeblair | clarkb: so there are no logs from that. | 22:16 |
| jeblair | clarkb: xinetd connection limit? | 22:16 |
| BobBall_Away | sorry - set python-path=.../site-packages | 22:16 |
| clarkb | jeblair: yeah I am thinking xinetd may be the culprit | 22:16 |
| mordred | BobBall_Away: can you show me find $VIRTUAL_ENV -type d -name site-packages output? | 22:16 |
| fungi | 38177 lgtm, assuming we've decided a) puppet vcsrepo should use https:// instead of git:// and clone urls should consistently omit .git suffixes. i'm ambivalent on both counts. also i guess replacing the github-org parameter in jjb config will be a separate change? | 22:16 |
| clarkb | jeblair: unable to fork off git daemons quick enough | 22:16 |
| mordred | BobBall_Away: (just so I can be smarter) | 22:16 |
| BobBall_Away | mordred: two lines: /opt/stack/new/.venv/lib/python2.7/site-packages | 22:16 |
| BobBall_Away | /opt/stack/new/.venv/lib/python2.7/site-packages/flask/testsuite/test_apps/lib/python2.5/site-packages | 22:17 |
| mordred | o m g | 22:17 |
| mordred | you're kidding me | 22:17 |
| BobBall_Away | mordred: although it works in my env, that's what the gate sees | 22:17 |
| mordred | ok | 22:17 |
| BobBall_Away | *grin* | 22:17 |
| BobBall_Away | I wish. | 22:17 |
| *** mriedem has quit IRC | 22:17 | |
| BobBall_Away | could restrict the depth with find easily, but I just thought that we might be able to rely on .venv/lib/python*/site-packages | 22:17 |
| *** qs201 has joined #openstack-infra | 22:18 | |
| BobBall_Away | (btw - if you're crafting a new patch, can you also fix lib/nova to execute get_python_bin_prefix rather than using that string? ;) ) | 22:18 |
| mordred | BobBall_Away: yeah - -mindepth 3 -maxdepth 3 should work | 22:18 |
| clarkb | jeblair: yeah looks like our xinetd.conf sets some conservative defaults. | 22:18 |
| BobBall_Away | ok | 22:18 |
| mordred | BobBall_Away: but also the * should work | 22:18 |
| mordred | BobBall_Away: and go for it | 22:18 |
| clarkb | jeblair: we can just update those numbers for git daemon | 22:18 |
| BobBall_Away | I'll upload a patch | 22:19 |
| BobBall_Away | ta! | 22:19 |
| mordred | BobBall_Away: I'm not crafting one :) | 22:19 |
| *** xBsd has quit IRC | 22:19 | |
| mordred | BobBall_Away: oh - and HAHAHA on the get_python_bin_prefix | 22:19 |
| BobBall_Away | One will be there before you can type shakespeare's hamlet backwards | 22:19 |
| *** qs201 has quit IRC | 22:19 | |
| jeblair | clarkb: ok; i'm going to see if i can trigger the error; you want to look up some new values? | 22:19 |
| clarkb | jeblair: yup reading man pages now | 22:19 |
| *** datsun180b has quit IRC | 22:20 | |
| jeblair | clarkb: i think i ran into per_source= 10 | 22:20 |
| jeblair | clarkb: got a connection reset by peer if i tried to clone more than 10 repos at once | 22:21 |
| jeblair | from a single host | 22:21 |
| clarkb | yup that is one of them | 22:21 |
| jeblair | clarkb: that one maybe we ought to leave alone? | 22:21 |
| jeblair | we don't actually do that anywhere | 22:21 |
| *** pabelanger has joined #openstack-infra | 22:23 | |
| *** dina_belova has joined #openstack-infra | 22:24 | |
| jeblair | clarkb: yeah, i can get the same error if i do > 50 clones from different hosts | 22:24 |
| *** prad_ has quit IRC | 22:24 | |
| clarkb | jeblair: yup and then there is a 10 second timeout | 22:24 |
| *** changbl has quit IRC | 22:24 | |
| *** jhesketh has joined #openstack-infra | 22:25 | |
| clarkb | jeblair: the settings is 'cps' trying to figure out if we need to give it numeric values or if we can say UNLIMITED 0 | 22:25 |
| clarkb | or I can try bumping to 16k or something high like that | 22:26 |
| *** rcleere has quit IRC | 22:28 | |
| jeblair | clarkb: why don't we say something like 500 10 ? | 22:28 |
| clarkb | jeblair: ok, I think I want to drop the timeout slightly but we can try that | 22:28 |
| jeblair | clarkb: even with a very large gate reset, we shouldn't actually hit that. | 22:29 |
| *** dina_belova has quit IRC | 22:29 | |
| jeblair | i wish we had a real socket queue here... | 22:29 |
| *** jerryz has quit IRC | 22:30 | |
| jeblair | because the server does not, at the moment, actually perform that well with a lot of instances running. | 22:30 |
| jeblair | but i think that's what we'd need to change in order to avoid the error we saw | 22:31 |
| anteaya | mordred: I have sent you a pm | 22:31 |
| anteaya | if you are around | 22:31 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Relax cgit xinetd ACLs. https://review.openstack.org/42773 | 22:31 |
| clarkb | jeblair: ^ | 22:31 |
| jeblair | clarkb: you sure about unlimited? | 22:32 |
| jeblair | is there another way to run a git daemon other than xinetd? | 22:32 |
| *** eharney has quit IRC | 22:32 | |
| jeblair | clarkb: because with the current tuning, the server got rather heavily loaded earlier | 22:33 |
| *** ryanpetrello has quit IRC | 22:33 | |
| *** dina_belova has joined #openstack-infra | 22:34 | |
| lifeless | hey question on the pypi.o.o mirror | 22:37 |
| lifeless | is there a mirror at / or just at /openstack/ ? | 22:37 |
| *** UtahDave has quit IRC | 22:37 | |
| lifeless | https://review.openstack.org/#/c/42615/ for context | 22:37 |
| jeblair | lifeless: the openstack/ mirror is at /openstack; other mirrors may exist later (eg openstack-infra) | 22:37 |
| lifeless | jeblair: ok, but they would be siblings, yeah ? | 22:37 |
| jeblair | lifeless: yes, in fact there is a stub there: http://pypi.openstack.org/ | 22:38 |
| lifeless | jeblair: right, it /looks/ like a mirror but it's empty | 22:38 |
| clarkb | jeblair: I am not sure about UNLIMITEd and yes I think we can run it as a normal service if we add an init script (this is how ubuntu's packaging does it) | 22:38 |
| lifeless | we've got something accidentally using / and breaking | 22:38 |
| *** chuckieb|2 has joined #openstack-infra | 22:39 | |
| jeblair | lifeless: i believe that patch dtrt | 22:39 |
| *** dina_belova has quit IRC | 22:39 | |
| jeblair | clarkb: i doubt running it as its own servic would help either, i don't see anything about socket queue depth in the params | 22:40 |
| clarkb | jeblair: --max-connections=32 is the default for git-daemon | 22:41 |
| *** chuckieb has quit IRC | 22:42 | |
| clarkb | jeblair: I can change the UNLIMITED to 1024 to give us some room to have more connections running concurrently without going overboard | 22:42 |
| jeblair | clarkb: that's just it, i'm not sure 50 is wrong | 22:42 |
| jeblair | clarkb: the server can not handle 50 connections very well | 22:43 |
| clarkb | I see | 22:43 |
| jeblair | clarkb: i think we should switch the tests to https:// and i think someone needs to do some serious thinking and testing about git:// | 22:43 |
| clarkb | jeblair: wfm | 22:43 |
| jeblair | clarkb: so we don't just dos ourselves (or invite someone else to do it for us) | 22:44 |
| mordred | ++ | 22:46 |
| clarkb | jeblair: I think our jenkins setup if a giant dos machine :) | 22:46 |
| clarkb | grenade is passing again | 22:47 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use https://git.openstack.org in jobs https://review.openstack.org/42777 | 22:47 |
| *** pcm_ has quit IRC | 22:49 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use git.openstack.org as origin https://review.openstack.org/42693 | 22:49 |
| jeblair | i think az1 has started responding | 22:49 |
| jeblair | i also think nodepool is leaking nodes, so i'll need to work on that for a bit. | 22:51 |
| clarkb | jeblair: ok | 22:51 |
| mordred | clarkb, jeblair: both the above look great - should I aprv or wait? | 22:52 |
| mordred | nm | 22:52 |
| mordred | :) | 22:52 |
| clarkb | mordred: I think you can approve | 22:53 |
| mordred | clarkb: jeblair got one of them already | 22:53 |
| jeblair | mordred: i could go either way on the other; no rush | 22:53 |
| mordred | kk | 22:53 |
| jeblair | the one i approved was to change git:// to https, the other was still using review https | 22:54 |
| mordred | yah | 22:54 |
| mordred | clarkb: do we need to abandon this one now? https://review.openstack.org/#/c/42773/ | 22:55 |
| jeblair | mordred, clarkb: i'm seeing more rate limit errors from az1 than i would expect (and more than we're getting for the other azs) | 22:55 |
| jeblair | mordred, clarkb: any chance something changed there? | 22:55 |
| mordred | jeblair: it's possible that they ratched down rate limit to get themselves out of the hole? | 22:56 |
| mordred | jeblair: but I do not know | 22:56 |
| mordred | lemme pop onto IM and see | 22:56 |
| *** weshay has joined #openstack-infra | 22:57 | |
| lifeless | jeblair: mordred: do you want a review from me on the dib using d-g patch? If it's going to be moved forward this week I'll happily do so. | 22:58 |
| pleia2 | tbh I didn't put much thought into git daemon while testing - I assumed it was a proven robust thing :\ | 22:58 |
| *** rnirmal has quit IRC | 22:58 | |
| clarkb | pleia2: I think it is robust, the problem here is that it is able to quickly dos the server. | 22:59 |
| pleia2 | heh, right | 22:59 |
| *** emagana has quit IRC | 22:59 | |
| *** nati_ueno has quit IRC | 22:59 | |
| clarkb | jeblair: pleia2: I am not entirely concinced that apache doens't DOS us in the same manner though | 22:59 |
| clarkb | the load on review.o.o was ridiculous and almost purely from apache | 22:59 |
| mordred | clarkb: I think eventually we're going to have to horizontally scale those servers | 23:00 |
| clarkb | mordred: I agree | 23:00 |
| jeblair | clarkb: apache has a socket queue, with appropriate tuning you can get reasonable load without immediate rejecting new connections | 23:00 |
| mordred | jeblair: ++ | 23:00 |
| notmyname | is http://logs.openstack.org/14/38214/3/check/gate-swift-pep8/4ddaea7/console.html saying that the pep8 command couldn't be run, or that the pep8 check failed? | 23:00 |
| *** nati_ueno has joined #openstack-infra | 23:00 | |
| pleia2 | scaling git servers will be easy enough I think | 23:00 |
| mordred | notmyname: command failed | 23:00 |
| jeblair | and yes, we should scale it horizontally, but we'd be in a better position to do so once we get one of them the way we want. | 23:00 |
| mordred | notmyname: ./setup.py:1:1: H802 git commit title ('In the object replicator, make the length of certain lines logged configurable') should be under 50 chars | 23:00 |
| clarkb | jeblair: ++ | 23:00 |
| notmyname | mordred: ok, I saw that, but "InvocationError" sure sounds like something else. thanks. | 23:01 |
| clarkb | mordred: let it be known that using flake8/hacking to enforce commit message stuff is broken with the way people use git | 23:01 |
| Alex_Gaynor | hmm, so zuul seems to have a 92 event queue, why are those being slow to process | 23:01 |
| mordred | notmyname: agree | 23:01 |
| pleia2 | I need to run out for a bit, will be back in a few hours | 23:01 |
| mordred | clarkb: agree | 23:01 |
| mordred | clarkb: I'm on board with re-thinking how we deal with that | 23:01 |
| mordred | I think the sequencing is wrong | 23:02 |
| clarkb | Alex_Gaynor: I think all of our slaves are being used | 23:02 |
| clarkb | Alex_Gaynor: the gate queue gets priority so having a large number of changes there can starve the check queue | 23:02 |
| jeblair | clarkb: 'event queue' | 23:02 |
| Alex_Gaynor | clarkb: I thought that count was counting things before they even went into gearman? | 23:03 |
| * notmyname would be in favor of renaming the "pep8" check to "formatting" or something more specific to what is actually being tested | 23:03 | |
| clarkb | jeblair: oh wrong queue | 23:03 |
| mordred | notmyname: I aslo agree with that | 23:03 |
| jeblair | notmyname: i would accept that patch. it's your chance to get involved in infra! | 23:03 |
| mordred | notmyname: in fact, it'sa low-priority TDL for me - but I think given the number of places where tox -epep8 is referenced, it might take a little while to phase in | 23:03 |
| mordred | ooh. I like jeblair's suggestion better | 23:03 |
| notmyname | mordred: clearly, sec | 23:03 |
| notmyname | sed | 23:04 |
| clarkb | Alex_Gaynor: jeblair: the event queue relies on git in its processing. Git is currently being DDoSed by our lovely slaves. | 23:04 |
| Alex_Gaynor | ah, fun times :) | 23:04 |
| mordred | notmyname: yeah - if we changed the name of the job in jenkins/zuul, that would be an excellent start | 23:04 |
| jeblair | clarkb: yeah, that's probably it. | 23:04 |
| mordred | eventually, I'd love to rename the tox environments too - but I don't think that's as visible | 23:04 |
| jeblair | mordred: i think they should go hand-in-hand. i'd hate to have that half-finished. | 23:05 |
| mordred | jeblair: yeah. although I think it's going to be a multi-phase thing | 23:05 |
| jeblair | mordred: yep, but no reason to approve the first phase before the second is at least staged. | 23:05 |
| mordred | jeblair: we're going to need run-pep8.sh to be able to see which env the project has (pep8 or formatting), then start changing all the names of all of the tox envs | 23:06 |
| jeblair | mordred: yep | 23:06 |
| clarkb | jeblair: pleia2 mordred if we put haproxy in front of xinetd/git-daemon we can use its socket queuing and that can evolve into using mutliple hosts to serve teh content | 23:06 |
| mordred | now THERE is a script folks will LOVE | 23:06 |
| * mordred submits 400 patches to gerrit in the same 15 minute span ... | 23:06 | |
| clarkb | jeblair: pleia2 mordred does that seem like a sane next step as we want to scale out horizontally? | 23:06 |
| notmyname | while we're talking of test failures, can anyone tell me why http://logs.openstack.org/14/38214/3/check/gate-grenade-devstack-vm/1a683be/console.html failed? | 23:06 |
| mordred | notmyname: yes, although it has since been fixed | 23:07 |
| mordred | notmyname: a backwards incompatible change to glanceclient got merged | 23:07 |
| notmyname | mordred: since 12:40 pm todya? | 23:07 |
| mordred | and we have not yet merged the change to start testing backwards incompatible changes | 23:07 |
| jeblair | clarkb: do we want to do our own load balancing, use a lbaas, or dns round robin? | 23:07 |
| mordred | notmyname: yes | 23:07 |
| notmyname | mordred: how did that happen? /me doesnt' really want to know, but seems like the think jenkins is supposed to prevent | 23:07 |
| mordred | notmyname: we're ALMOST ready with testing that trunk clients work with stable releases ... | 23:07 |
| clarkb | jeblair: I am wary of lbaas because everyone uses a different api iirc. mordred would know better | 23:08 |
| mordred | clarkb: I would want to use lbaas | 23:08 |
| *** mgagne has quit IRC | 23:08 | |
| clarkb | jeblair: dns round robin would be simplest but would not help with connection queueing | 23:08 |
| mordred | clarkb: rax and hp are actually using the same API | 23:08 |
| clarkb | mordred: but it is different than what openstack/neutron are doing right? | 23:08 |
| *** ^d has quit IRC | 23:08 | |
| jeblair | mordred: layer 7 or 2/3? | 23:08 |
| mordred | it is - but the plan (from both, aiui) is to move to openstack/neutron | 23:09 |
| mordred | jeblair: it's haproxy aaS | 23:09 |
| jeblair | clarkb: then perhaps we should start looking into that now | 23:10 |
| clarkb | jeblair: ya I have pulled up the rax docs | 23:10 |
| mordred | clarkb: I believe python-libraclient _should_ be able to work with it, since I think they implemented the same api | 23:11 |
| mordred | but Shrews would know more | 23:11 |
| Alex_Gaynor | clarkb, mordred: If ya'll have a second https://review.openstack.org/#/c/42692/ is fixed up now | 23:12 |
| mordred | jeblair, clarkb: also, given the nature of these - would it make any sense to have a set in rax and a set in hp? | 23:12 |
| *** eharney has joined #openstack-infra | 23:12 | |
| clarkb | mordred: then you have to do "global" load balancing of some sort (which may be as simple as a dns round robin) | 23:13 |
| clarkb | or hot standby | 23:13 |
| mordred | clarkb: yah. or also have global dns round robin _plus_ a local cloud affinity | 23:13 |
| mordred | so that the hpcloud slaves go to hpcloud git.o.o for their stuff or something? I dunno, might be too complex -and network doesn't seem to be the issue | 23:14 |
| *** weshay has quit IRC | 23:14 | |
| jeblair | mordred, clarkb: let's just try to get one system where we don't actually dos ourselves first. | 23:14 |
| mordred | jeblair: ++ | 23:14 |
| Alex_Gaynor | It seems to be a low of nature that any sufficiently interesting distributed system will DDoS itself at some point | 23:15 |
| jeblair | mordred, clarkb: i've manually tuned some nodepool values to slow it down a bit in az1 | 23:15 |
| mordred | jeblair: I love that Alex_Gaynor's patch is actually one of the things we were hoping we'd see from stackforge- someone adds a feature for a stackforge project that quite clearly could wind up being used by openstack projects too | 23:15 |
| Alex_Gaynor | mordred: what'd I do? | 23:16 |
| mordred | adding the pypy support | 23:16 |
| Alex_Gaynor | oh, yeah :) | 23:16 |
| jeblair | ++ | 23:16 |
| Alex_Gaynor | Once this lands I'll probably add it to a few clients as well | 23:16 |
| jeblair | i have to run now | 23:16 |
| openstackgerrit | A change was merged to openstack-infra/config: Use https://git.openstack.org in jobs https://review.openstack.org/42777 | 23:17 |
| clarkb | mordred: I am not seeing anything about connection queueing in the rax api | 23:17 |
| clarkb | mordred: there are throttles but it is unclear if throttling with NACK or queue | 23:18 |
| clarkb | s/with/will/ | 23:18 |
| clarkb | but it looks relatively straightforward to set one of these up, and we can do so without impacting git.o.o directly. | 23:18 |
| * clarkb goes to find pcrews and see if he knows what the throttling will do | 23:19 | |
| clarkb | mordred: jeblair http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/Throttle_Connections-d1e4057.html | 23:33 |
| *** sandywalsh has quit IRC | 23:34 | |
| clarkb | that is per source throttling and I don't read it as providing queuing | 23:34 |
| clarkb | something like http://code.google.com/p/haproxy-docs/wiki/maxconn is what we would want in front of git-daemon | 23:34 |
| *** dina_belova has joined #openstack-infra | 23:35 | |
| mordred | clarkb: I believe I agree with you | 23:35 |
| mordred | and also that that is what we'd want | 23:36 |
| mordred | sign | 23:36 |
| mordred | sigh | 23:36 |
| *** changbl has joined #openstack-infra | 23:36 | |
| mordred | maybe just running a local haproxy for connection throttling is the thing we want | 23:36 |
| mordred | otoh, maybe http/apache just solves it for us | 23:36 |
| clarkb | mordred: I think http/apache solves it for git over http. It may not be worth providing git:// as a result | 23:37 |
| mordred | yah | 23:37 |
| *** tjones has left #openstack-infra | 23:37 | |
| clarkb | though puppetlabs seems to provide a fairly robust haproxy module | 23:38 |
| *** dina_belova has quit IRC | 23:40 | |
| *** ^d has joined #openstack-infra | 23:41 | |
| *** ^d has joined #openstack-infra | 23:41 | |
| *** eharney has quit IRC | 23:41 | |
| SpamapS | hm, Is it possible a new quantumclient released today with an api change that might be breaking Heat's unit tests? | 23:43 |
| SpamapS | oh wait | 23:43 |
| SpamapS | neutron | 23:43 |
| SpamapS | I thought we already finished this? Argh. | 23:43 |
| SpamapS | stale pyc's .. doh | 23:45 |
| mordred | hahaha | 23:45 |
| * SpamapS invites everyone to ignore the previous 6 lines. | 23:45 | |
| mordred | SpamapS: I was already ignoring them | 23:46 |
| SpamapS | mordred: again, but this time with feeling | 23:46 |
| * mordred ignores the SHIT out of the previous 6 lines | 23:46 | |
| SpamapS | thank you | 23:47 |
| *** sandywalsh has joined #openstack-infra | 23:49 | |
| *** mrodden has joined #openstack-infra | 23:50 | |
| jeblair | mordred, clarkb: it may be time for us to chat with kernel.org or fedora infra people about how to host git at scale. | 23:54 |
| mordred | jeblair: do they host systems that want to clone this much? | 23:54 |
| mordred | jeblair: and yes | 23:55 |
| mordred | I believe it probably is | 23:55 |
| clarkb | jlk: ^ | 23:57 |
| jlk | oh hai | 23:59 |
| jlk | I setup Fedora's git server a while back | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!