| *** nmagnezi has quit IRC | 00:01 | |
| jhesketh | clarkb: If you're still around, some help moving https://review.openstack.org/#/q/status:open+topic:enable_swift along this week would be great | 00:04 |
|---|---|---|
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: (WIP) Upload metadata for pseudo folder https://review.openstack.org/263240 | 00:06 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Send instructions to workers to upload metadata https://review.openstack.org/263218 | 00:06 |
| *** aeng has joined #openstack-infra | 00:07 | |
| *** aeng has quit IRC | 00:25 | |
| *** mudassirlatif has joined #openstack-infra | 00:25 | |
| daemontool | clarkb, that was part of the issue, thank you | 00:26 |
| daemontool | : ) | 00:27 |
| *** yamamoto has quit IRC | 00:27 | |
| *** Jeffrey4l has quit IRC | 00:28 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 00:29 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add a helper function to convert xml https://review.openstack.org/261620 | 00:29 |
| *** mudassirlatif has quit IRC | 00:37 | |
| *** dims has joined #openstack-infra | 00:39 | |
| *** gmann has joined #openstack-infra | 00:43 | |
| *** thorst has joined #openstack-infra | 00:47 | |
| *** thorst has quit IRC | 00:48 | |
| *** thorst has joined #openstack-infra | 00:49 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin https://review.openstack.org/261587 | 00:51 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add helper function to convert xml https://review.openstack.org/261620 | 00:51 |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Remove unused setUp condition https://review.openstack.org/268825 | 00:51 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: bump timeout for tempest-dsvm-keystone-eventlet-full https://review.openstack.org/268826 | 00:53 |
| stevemar | something wrong with: http://logstash.openstack.org/#/dashboard ? | 00:53 |
| *** thorst has quit IRC | 00:53 | |
| *** dims has quit IRC | 01:01 | |
| *** otter768 has joined #openstack-infra | 01:03 | |
| *** sabeen3 has joined #openstack-infra | 01:03 | |
| *** davidlenwell has quit IRC | 01:05 | |
| *** sabeen has quit IRC | 01:05 | |
| *** salv-orlando has quit IRC | 01:06 | |
| *** davideagnello has joined #openstack-infra | 01:06 | |
| *** salv-orlando has joined #openstack-infra | 01:07 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Omit subunit2sql/tests from coverage https://review.openstack.org/268831 | 01:07 |
| *** otter768 has quit IRC | 01:07 | |
| *** davidlenwell has joined #openstack-infra | 01:07 | |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Added Test Selector Plugin support https://review.openstack.org/268795 | 01:08 |
| *** tlian2 has joined #openstack-infra | 01:10 | |
| *** tlian has quit IRC | 01:11 | |
| *** davideagnello has quit IRC | 01:12 | |
| *** bardia has quit IRC | 01:12 | |
| *** lerty_ has joined #openstack-infra | 01:12 | |
| *** lerty_ has quit IRC | 01:12 | |
| *** dimtruck is now known as zz_dimtruck | 01:15 | |
| *** lerty has joined #openstack-infra | 01:17 | |
| *** lerty has left #openstack-infra | 01:17 | |
| *** rfolco has joined #openstack-infra | 01:18 | |
| *** rfolco has quit IRC | 01:18 | |
| *** baoli has quit IRC | 01:20 | |
| *** dingyichen has quit IRC | 01:23 | |
| *** fedexo has joined #openstack-infra | 01:25 | |
| *** aeng has joined #openstack-infra | 01:26 | |
| *** tjones has quit IRC | 01:26 | |
| *** sonali5 has joined #openstack-infra | 01:29 | |
| *** amotoki has joined #openstack-infra | 01:30 | |
| *** amotoki has quit IRC | 01:32 | |
| *** baoli has joined #openstack-infra | 01:33 | |
| *** Daisy has joined #openstack-infra | 01:33 | |
| *** MarkAtwood has quit IRC | 01:34 | |
| openstackgerrit | Merged openstack/requirements: Updated from global requirements https://review.openstack.org/268532 | 01:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/268438 | 01:41 |
| *** yamamoto_ has joined #openstack-infra | 01:42 | |
| *** MarkAtwood has joined #openstack-infra | 01:45 | |
| *** bpokorny has joined #openstack-infra | 01:46 | |
| *** miqui has quit IRC | 01:49 | |
| *** amotoki has joined #openstack-infra | 01:50 | |
| *** zhurong has joined #openstack-infra | 01:50 | |
| *** thorst has joined #openstack-infra | 01:51 | |
| *** Sukhdev has joined #openstack-infra | 01:51 | |
| *** MarkAtwood has quit IRC | 01:57 | |
| *** thorst has quit IRC | 01:58 | |
| *** changzhi has joined #openstack-infra | 02:02 | |
| *** amotoki has quit IRC | 02:02 | |
| *** yuanying has quit IRC | 02:04 | |
| *** sonali5 has quit IRC | 02:05 | |
| *** yuanying has joined #openstack-infra | 02:05 | |
| *** MarkAtwood has joined #openstack-infra | 02:05 | |
| *** Qiming has joined #openstack-infra | 02:06 | |
| *** mriedem has quit IRC | 02:06 | |
| *** Jeffrey4l has joined #openstack-infra | 02:07 | |
| *** Jeffrey4l has quit IRC | 02:07 | |
| *** sonali5 has joined #openstack-infra | 02:07 | |
| *** Jeffrey4l has joined #openstack-infra | 02:07 | |
| *** aeng has quit IRC | 02:10 | |
| *** binbincong has joined #openstack-infra | 02:13 | |
| *** jamielennox is now known as jamielennox|away | 02:14 | |
| *** yamahata has joined #openstack-infra | 02:15 | |
| *** zz_dimtruck is now known as dimtruck | 02:20 | |
| *** yamahata has quit IRC | 02:22 | |
| openstackgerrit | IWAMOTO Toshihiro proposed openstack/requirements: Bump ryu to 3.29.1 https://review.openstack.org/264026 | 02:24 |
| *** PsionTheory has quit IRC | 02:24 | |
| *** markvoelker_ has quit IRC | 02:26 | |
| *** fedexo has quit IRC | 02:27 | |
| *** bpokorny has quit IRC | 02:29 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Prune old branchs in early devstack clone https://review.openstack.org/268841 | 02:38 |
| ianw | clarkb: ^ i think that is the right fix. super confusing when you're grepping for logs/commands, and they show up in DIB so you think you're fixing it in the right place! | 02:38 |
| *** MarkAtwood has quit IRC | 02:39 | |
| *** binbincong has quit IRC | 02:39 | |
| *** sonali5 has quit IRC | 02:41 | |
| *** pahuang has joined #openstack-infra | 02:42 | |
| *** dingyichen has joined #openstack-infra | 02:44 | |
| *** sonali5 has joined #openstack-infra | 02:44 | |
| *** hdd has joined #openstack-infra | 02:45 | |
| *** eliqiao_ has joined #openstack-infra | 02:47 | |
| *** amotoki has joined #openstack-infra | 02:48 | |
| *** bpokorny has joined #openstack-infra | 02:48 | |
| *** eliqiao has quit IRC | 02:50 | |
| *** sonuk has joined #openstack-infra | 02:51 | |
| *** baoli has quit IRC | 02:51 | |
| *** dims has joined #openstack-infra | 02:52 | |
| *** davideagnello has joined #openstack-infra | 02:54 | |
| *** davideagnello has quit IRC | 02:54 | |
| *** thorst has joined #openstack-infra | 02:56 | |
| *** binbincong has joined #openstack-infra | 02:56 | |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Added Test Selector Plugin support https://review.openstack.org/268795 | 02:57 |
| *** rguillebert has quit IRC | 02:57 | |
| *** baoli has joined #openstack-infra | 02:59 | |
| *** yuanying has quit IRC | 03:00 | |
| *** yuanying has joined #openstack-infra | 03:01 | |
| *** baoli has quit IRC | 03:01 | |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Added Test Selector Plugin support https://review.openstack.org/268795 | 03:01 |
| *** thorst has quit IRC | 03:03 | |
| *** otter768 has joined #openstack-infra | 03:04 | |
| *** nikhil has quit IRC | 03:04 | |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support https://review.openstack.org/268795 | 03:05 |
| *** nikhil has joined #openstack-infra | 03:05 | |
| *** aeng has joined #openstack-infra | 03:07 | |
| *** harlowja_at_home has joined #openstack-infra | 03:08 | |
| *** otter768 has quit IRC | 03:08 | |
| *** sonuk has quit IRC | 03:09 | |
| *** pal has joined #openstack-infra | 03:10 | |
| *** pal is now known as Guest43844 | 03:11 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 03:12 |
| *** mudassirlatif has joined #openstack-infra | 03:14 | |
| *** yuanying has quit IRC | 03:20 | |
| *** dimtruck is now known as zz_dimtruck | 03:20 | |
| *** Jeffrey4l has quit IRC | 03:21 | |
| *** mudassirlatif has quit IRC | 03:23 | |
| *** mudassirlatif has joined #openstack-infra | 03:25 | |
| *** links has joined #openstack-infra | 03:26 | |
| *** bpokorny has quit IRC | 03:27 | |
| *** baoli has joined #openstack-infra | 03:35 | |
| *** Jeffrey4l has joined #openstack-infra | 03:35 | |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support https://review.openstack.org/268795 | 03:36 |
| *** Daisy has quit IRC | 03:37 | |
| *** ddieterly has quit IRC | 03:37 | |
| *** baoli has quit IRC | 03:39 | |
| *** mudassirlatif has quit IRC | 03:40 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 03:42 |
| *** Guest43844 has quit IRC | 03:42 | |
| *** sonali5 has quit IRC | 03:43 | |
| *** salv-orl_ has joined #openstack-infra | 03:45 | |
| *** salv-orlando has quit IRC | 03:47 | |
| stevemar | so... http://logstash.openstack.org/#/dashboard is very different since the last time i looked at it, any tips on where the search field is? :\ | 03:48 |
| stevemar | just seeing a black background | 03:48 |
| openstackgerrit | ke liang proposed openstack/requirements: Bump oslo.messaging version to 4.0.0 https://review.openstack.org/268856 | 03:48 |
| *** Sukhdev has quit IRC | 03:51 | |
| *** Liuqing has joined #openstack-infra | 03:54 | |
| *** flwang has quit IRC | 03:56 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add non-voting rally job https://review.openstack.org/268859 | 04:00 |
| *** thorst has joined #openstack-infra | 04:01 | |
| *** bpokorny has joined #openstack-infra | 04:01 | |
| *** watanabe_isao has joined #openstack-infra | 04:02 | |
| *** treaki_ has joined #openstack-infra | 04:03 | |
| watanabe_isao | hello all, I have meat a problem that dib cannot give me a image when I built a CI and run <nodepool image-build> command for the first time. | 04:03 |
| watanabe_isao | There is only two lines of log output, and the command waits there forever (at lease 48+ hours) | 04:05 |
| watanabe_isao | Have anyone meet the same problem, or have a solution, please? | 04:06 |
| *** yuanying has joined #openstack-infra | 04:06 | |
| *** treaki has quit IRC | 04:07 | |
| Thelo | watanabe_isao: can you show the log s? | 04:07 |
| *** thorst has quit IRC | 04:08 | |
| watanabe_isao | Thelo, yes. | 04:08 |
| *** kushal has joined #openstack-infra | 04:08 | |
| watanabe_isao | 2016-01-18 13:02:43,804 INFO gear.Connection.nodepool: Sending packet to <gear.Connection 0x7fd8cf0ca710 host: localhost port: 4730>: <gear.Packet 0x7fd8cdd48890 type: SUBMIT_JOB function: image-build:image unique: aa194908dec1428fa7e37b528b6a189f> | 04:08 |
| watanabe_isao | 2016-01-18 13:02:43,805 INFO gear.Client.nodepool: Received packet from <gear.Connection 0x7fd8cf0ca710 host: localhost port: 4730>: <gear.Packet 0x7fd8cdd6b0d0 type: JOB_CREATED handle: H:127.0.0.1:8> | 04:08 |
| *** baoli has joined #openstack-infra | 04:10 | |
| *** baoli has quit IRC | 04:11 | |
| Thelo | watanabe_isao: where do you seen this log? | 04:12 |
| Thelo | seen/see* | 04:12 |
| greghaynes | watanabe_isao: are you running a nodepool daemon? | 04:12 |
| watanabe_isao | Thelo, the log is the output of <nodepool image-build> command | 04:12 |
| watanabe_isao | greghaynes, no as the procedure of <https://github.com/openstack-infra/puppet-openstackci/blob/master/contrib/README.md> I think I should start nodepool after the image has been built. | 04:13 |
| *** bpokorny has quit IRC | 04:14 | |
| greghaynes | watanabe_isao: seems like you arent - you need to be runnign an image builder daemon in order to build images now. You can do this but running the nodepool-builder command or running a full nodepool daemon | 04:14 |
| Thelo | watanabe_isao: it looks like nodepool's log rather than the image-build output : in my machine the image-build about console output looks like :"2016-01-18 00:00:00,086 INFO nodepool.image.build.dpc: + true" | 04:14 |
| greghaynes | watanabe_isao: looks like those docs are out of date. There was a recent change to how nodepool works which requires this now | 04:15 |
| greghaynes | Thelo: We recently merged distributed builders which changed that | 04:15 |
| *** Sukhdev has joined #openstack-infra | 04:16 | |
| Thelo | greghaynes: Could that affect the CI who used the previous method? in my case, since yesterday night the image built by nodepool does not boot properly | 04:16 |
| greghaynes | Thelo: This would not cause that - sounds like a dib issue | 04:17 |
| watanabe_isao | greghaynes, thanks for telling me this. After I refreshed my browser I found the newest message. I'll have a look at this. | 04:18 |
| Thelo | greghaynes: yes, however I couldn't find any ERROR log in the trace ( except that : INFO nodepool.image.build.dpc: sfdisk: ERROR: sector 0 does not have an msdos signature , but I had it when it was working too ) | 04:19 |
| watanabe_isao | greghaynes, Thelo, thank you very much. | 04:19 |
| greghaynes | Thelo: Ya, there were some recent changes to the dib bootloader element - what elements are you using in your images | 04:19 |
| greghaynes | Thelo: also, when you say they dont boot properly, how far in to the boot process do they get? | 04:19 |
| *** Daisy has joined #openstack-infra | 04:20 | |
| Thelo | greghaynes: they stuck after "random: nonblocking pool is initialized" | 04:21 |
| greghaynes | Thelo: oh, so not the bootloader then | 04:21 |
| greghaynes | Thelo: I dont really know what would cause that | 04:22 |
| Thelo | greghaynes: me too | 04:22 |
| Thelo | is there anyway to force nodepool to use yesterday's image? | 04:22 |
| greghaynes | delete the newer image | 04:22 |
| *** fawadkhaliq has joined #openstack-infra | 04:23 | |
| Thelo | greghaynes: in nodepool directly? | 04:23 |
| *** baoli has joined #openstack-infra | 04:23 | |
| greghaynes | using nodepool, yes | 04:23 |
| *** otter768 has joined #openstack-infra | 04:24 | |
| *** maishsk has quit IRC | 04:25 | |
| *** hichihara has joined #openstack-infra | 04:26 | |
| Thelo | greghaynes: should it take time? | 04:27 |
| watanabe_isao | BTW, has anyone met the "slave went offline" issue before? That is as soon as a slave is done with its job and been deleted by jenkins, other slaves will went offline whatever how good progress the job runs. | 04:29 |
| greghaynes | Thelo: new nodes should use the next older image | 04:29 |
| *** vivekd has joined #openstack-infra | 04:29 | |
| *** oomichi has joined #openstack-infra | 04:31 | |
| Thelo | greghaynes: The command dib-image-delete has terminated yet, so the new nodes keeps using this image. | 04:32 |
| watanabe_isao | Thelo, I think you should use <nodepool image-list> to get the image ids, and use <nodepool image-delete <newer-id>> to delete the today's image from provider. If I did not get greghaynes wrong. | 04:34 |
| greghaynes | yes, sorry, use nodepool-image-delete to delete the image uploaded to the provider, that is what your nodes boot off of | 04:35 |
| *** otter768 has quit IRC | 04:35 | |
| *** vivekd has quit IRC | 04:36 | |
| Thelo | grehaynes, watanabe_isao : thanks | 04:36 |
| watanabe_isao | greghaynes, Thelo, I forgot to added your name. Do you have any idea about my last question of <slave went offline> issue? | 04:37 |
| greghaynes | I dont know much about the jenkins interaction, sorry | 04:38 |
| *** ddieterly has joined #openstack-infra | 04:39 | |
| *** maishsk has joined #openstack-infra | 04:41 | |
| Thelo | greghaynes: once in a while I get this error : FATAL: no longer a configured node for d-p-c-local_01-769 in my job's console | 04:43 |
| Thelo | and the job fails | 04:43 |
| Thelo | do you know the reason? | 04:43 |
| Thelo | and how to avoid that? | 04:43 |
| *** ddieterly has quit IRC | 04:44 | |
| *** fawadkhaliq has quit IRC | 04:44 | |
| *** krtaylor has joined #openstack-infra | 04:45 | |
| *** maishsk has quit IRC | 04:47 | |
| *** maishsk has joined #openstack-infra | 04:48 | |
| watanabe_isao | Thelo, seems like we are facing different problems. Mine is <ERROR: Connection was broken: java.io.IOException: Unexpected termination of the channel>, and I have not figure it out yet. | 04:48 |
| *** sflanigan has joined #openstack-infra | 04:49 | |
| *** sflanigan has joined #openstack-infra | 04:49 | |
| *** oomichi has quit IRC | 04:50 | |
| *** ashishb has joined #openstack-infra | 04:52 | |
| *** sflanigan has quit IRC | 04:54 | |
| *** maishsk has quit IRC | 04:55 | |
| *** ashishb has quit IRC | 04:56 | |
| *** dims has quit IRC | 04:58 | |
| *** larainema has quit IRC | 04:58 | |
| *** pal has joined #openstack-infra | 04:58 | |
| *** pal is now known as Guest67563 | 04:59 | |
| *** Guest67563 has quit IRC | 04:59 | |
| *** thorst has joined #openstack-infra | 05:05 | |
| *** sflanigan has joined #openstack-infra | 05:08 | |
| *** sflanigan has joined #openstack-infra | 05:08 | |
| *** coolsvap|away is now known as coolsvap | 05:11 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Handle dib build failures https://review.openstack.org/268867 | 05:11 |
| *** harlowja_at_home has quit IRC | 05:12 | |
| *** aopincar has joined #openstack-infra | 05:13 | |
| *** thorst has quit IRC | 05:13 | |
| *** baoli has quit IRC | 05:17 | |
| *** salv-orl_ has quit IRC | 05:20 | |
| *** salv-orlando has joined #openstack-infra | 05:20 | |
| *** daemontool has quit IRC | 05:27 | |
| *** daemontool has joined #openstack-infra | 05:27 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump versions for each keystone library https://review.openstack.org/266629 | 05:29 |
| *** bardia has joined #openstack-infra | 05:30 | |
| *** daemontool has quit IRC | 05:33 | |
| *** fawadkhaliq has joined #openstack-infra | 05:34 | |
| *** sabeen3 has quit IRC | 05:37 | |
| *** mudassirlatif has joined #openstack-infra | 05:37 | |
| *** ddieterly has joined #openstack-infra | 05:40 | |
| *** jaosorior has joined #openstack-infra | 05:41 | |
| *** salv-orlando has quit IRC | 05:41 | |
| *** salv-orlando has joined #openstack-infra | 05:42 | |
| *** maishsk has joined #openstack-infra | 05:43 | |
| *** ddieterly has quit IRC | 05:45 | |
| *** isaacb has quit IRC | 05:46 | |
| *** tlian2 has quit IRC | 05:47 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups https://review.openstack.org/268404 | 05:52 |
| *** _et_ has joined #openstack-infra | 05:54 | |
| *** abregman has joined #openstack-infra | 05:56 | |
| *** Watanabe has joined #openstack-infra | 05:59 | |
| *** watanabe_isao has quit IRC | 05:59 | |
| *** maishsk has quit IRC | 06:08 | |
| *** thorst has joined #openstack-infra | 06:10 | |
| *** aeng has quit IRC | 06:16 | |
| *** thorst has quit IRC | 06:18 | |
| *** sputnik13 has quit IRC | 06:20 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 06:25 |
| *** ashishb has joined #openstack-infra | 06:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Cleanup the early devstack clone script https://review.openstack.org/268888 | 06:27 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Prune old branches in early devstack clone https://review.openstack.org/268841 | 06:27 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Cleanup the early devstack clone script https://review.openstack.org/268888 | 06:27 |
| *** sonali5 has joined #openstack-infra | 06:29 | |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support https://review.openstack.org/268795 | 06:31 |
| stevemar | AJaeger: o/ | 06:31 |
| stevemar | AJaeger: good morning! | 06:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/267962 | 06:32 |
| stevemar | AJaeger: any reason why http://logstash.openstack.org/#/dashboard is not loading a dashboard for me :) | 06:33 |
| *** otter768 has joined #openstack-infra | 06:36 | |
| *** sabeen1 has joined #openstack-infra | 06:36 | |
| *** sabeen2 has joined #openstack-infra | 06:38 | |
| *** yfried_ has quit IRC | 06:38 | |
| *** sabeen1 has quit IRC | 06:40 | |
| *** ddieterly has joined #openstack-infra | 06:41 | |
| *** otter768 has quit IRC | 06:41 | |
| *** isaacb has joined #openstack-infra | 06:45 | |
| *** ddieterly has quit IRC | 06:45 | |
| *** Ramanjaneya has joined #openstack-infra | 06:46 | |
| *** nikil89_ has joined #openstack-infra | 06:47 | |
| *** nikil89_ is now known as naggappan | 06:47 | |
| *** salv-orlando has quit IRC | 06:54 | |
| *** salv-orlando has joined #openstack-infra | 06:55 | |
| *** _nadya_ has joined #openstack-infra | 06:56 | |
| *** aeng has joined #openstack-infra | 06:58 | |
| *** chlong has quit IRC | 07:00 | |
| *** Qiming has quit IRC | 07:00 | |
| *** Qiming_ has joined #openstack-infra | 07:00 | |
| *** aeng has quit IRC | 07:01 | |
| openstackgerrit | Isaac Beckman proposed openstack/diskimage-builder: Move speedup section to image building guide https://review.openstack.org/268769 | 07:02 |
| *** salv-orlando has quit IRC | 07:03 | |
| *** salv-orlando has joined #openstack-infra | 07:04 | |
| *** mudassirlatif has quit IRC | 07:11 | |
| *** isaacb has quit IRC | 07:12 | |
| *** maishsk has joined #openstack-infra | 07:15 | |
| *** thorst has joined #openstack-infra | 07:16 | |
| *** coolsvap is now known as coolsvap|away | 07:18 | |
| *** gildub has quit IRC | 07:20 | |
| *** ifarkas has joined #openstack-infra | 07:22 | |
| *** thorst has quit IRC | 07:23 | |
| *** e0ne has joined #openstack-infra | 07:25 | |
| *** kushal has quit IRC | 07:26 | |
| *** sabeen2 has quit IRC | 07:26 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 07:27 |
| *** baoli has joined #openstack-infra | 07:29 | |
| *** larainema has joined #openstack-infra | 07:29 | |
| *** kushal has joined #openstack-infra | 07:29 | |
| *** baoli has quit IRC | 07:33 | |
| *** Qiming_ has quit IRC | 07:36 | |
| *** nmagnezi has joined #openstack-infra | 07:38 | |
| *** mpavone has joined #openstack-infra | 07:40 | |
| *** rcernin has joined #openstack-infra | 07:41 | |
| *** Qiming has joined #openstack-infra | 07:42 | |
| *** ddieterly has joined #openstack-infra | 07:42 | |
| *** dguitarbite has joined #openstack-infra | 07:45 | |
| *** salv-orlando has quit IRC | 07:46 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 07:46 |
| *** ddieterly has quit IRC | 07:46 | |
| *** jtomasek has joined #openstack-infra | 07:46 | |
| *** maishsk has quit IRC | 07:46 | |
| *** salv-orlando has joined #openstack-infra | 07:47 | |
| *** mudassirlatif has joined #openstack-infra | 07:48 | |
| *** daemontool has joined #openstack-infra | 07:48 | |
| *** isaacb has joined #openstack-infra | 07:50 | |
| *** _et_ has quit IRC | 07:51 | |
| *** bardia has quit IRC | 07:53 | |
| *** zeih has joined #openstack-infra | 07:54 | |
| openstackgerrit | afazekas proposed openstack-infra/system-config: Add ebtables to the default package set https://review.openstack.org/268906 | 07:55 |
| AJaeger | morning, stevemar . Sorry, can't help with logstash | 07:55 |
| *** esikachev has joined #openstack-infra | 07:55 | |
| esikachev | hey guys! can you review https://review.openstack.org/#/c/268145/? | 07:56 |
| *** daemontool has quit IRC | 07:58 | |
| *** _et_ has joined #openstack-infra | 07:58 | |
| *** yfried_ has joined #openstack-infra | 07:58 | |
| changzhi | hi all, could someone take a look at https://review.openstack.org/#/c/268847? i want check pep8 and unit test in jenkins job. | 08:02 |
| changzhi | but there is an error says "Job steth-pypi-both-upload not defined" | 08:03 |
| AJaeger | changzhi: did you see my comments on projects.yaml? | 08:05 |
| changzhi | AJaeger: yes, and i uploaded a new patch set. but the same error ;-( | 08:06 |
| *** salv-orlando has quit IRC | 08:06 | |
| AJaeger | changzhi: did you made the change I asked for? | 08:07 |
| changzhi | AJaeger: sure, i remove the line "gate-{name}-pep8" | 08:07 |
| *** salv-orlando has joined #openstack-infra | 08:08 | |
| AJaeger | but you did not add pypi-jobs | 08:08 |
| *** maishsk has joined #openstack-infra | 08:08 | |
| *** _nadya_ has quit IRC | 08:09 | |
| changzhi | AJaeger: should like http://paste.openstack.org/show/484101/ ? | 08:10 |
| AJaeger | changzhi: yes | 08:10 |
| *** matrohon has joined #openstack-infra | 08:11 | |
| AJaeger | and follow http://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases - looking at pypi, we do not have permissions to publish steth yet | 08:11 |
| changzhi | ok. what about zuul/layout.yaml? like this: http://paste.openstack.org/show/484102/? | 08:12 |
| *** fawadkhaliq has quit IRC | 08:12 | |
| *** salv-orlando has quit IRC | 08:13 | |
| *** fawadkhaliq has joined #openstack-infra | 08:13 | |
| AJaeger | remvoe the pre-release, it's part of publish-to-pypi | 08:13 |
| *** markus_z has joined #openstack-infra | 08:14 | |
| *** hichihara has quit IRC | 08:17 | |
| changzhi | AJaeger: ok. i have added openstackci permissions to steth just now. | 08:17 |
| *** hdd has quit IRC | 08:18 | |
| AJaeger | thanks | 08:18 |
| *** wei-wang has joined #openstack-infra | 08:18 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 08:19 |
| *** Jeffrey4l has quit IRC | 08:19 | |
| *** e0ne has quit IRC | 08:20 | |
| *** katyafervent is now known as katyafervent_awa | 08:21 | |
| *** lezbar has joined #openstack-infra | 08:21 | |
| *** thorst has joined #openstack-infra | 08:21 | |
| *** Daisy has quit IRC | 08:22 | |
| *** katyafervent_awa is now known as katyafervent | 08:22 | |
| asselin | Watanabe, slave going offline might be done explicitly. most are "single-use-slaves". | 08:22 |
| *** lezbar__ has joined #openstack-infra | 08:23 | |
| *** oomichi has joined #openstack-infra | 08:23 | |
| Watanabe | asselin, hello, yes I'm using it. | 08:23 |
| asselin | Watanabe, the " java.io.IOException: Unexpected termination" is probably a network issue | 08:23 |
| *** jcoufal has joined #openstack-infra | 08:24 | |
| asselin | Watanabe, also I noticed you're using the old repo from github. I encourage you to switch to this one which is community-supported: http://docs.openstack.org/infra/openstackci/third_party_ci.html | 08:25 |
| *** lezbar has quit IRC | 08:25 | |
| *** thorst has quit IRC | 08:27 | |
| Watanabe | asselin, sure. Thank you very much. BTW, If I want the node go offline as soon as the job is over, do you know what should I use instead of single-use-slaves, please? set_node_options? | 08:27 |
| *** [1]Thelo has joined #openstack-infra | 08:28 | |
| *** jaosorior has quit IRC | 08:29 | |
| *** sonali5 has quit IRC | 08:30 | |
| *** jaosorior has joined #openstack-infra | 08:30 | |
| *** Thelo has quit IRC | 08:30 | |
| *** [1]Thelo is now known as Thelo | 08:31 | |
| asselin | Watanabe, http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/openstack_functions.py#n32 | 08:32 |
| *** Jeffrey4l has joined #openstack-infra | 08:32 | |
| asselin | http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/zuul/layout.yaml#n67 | 08:32 |
| Watanabe | asselin, sorry I was wrong. I am using single_use_node | 08:34 |
| *** maishsk has quit IRC | 08:34 | |
| Watanabe | not single-use-slaves | 08:34 |
| *** fhubik has joined #openstack-infra | 08:35 | |
| *** yfried has joined #openstack-infra | 08:36 | |
| *** gildub has joined #openstack-infra | 08:36 | |
| *** yfried_ has quit IRC | 08:36 | |
| *** Jeffrey4l has quit IRC | 08:37 | |
| *** saggi has quit IRC | 08:37 | |
| *** otter768 has joined #openstack-infra | 08:37 | |
| *** samuelBartel has joined #openstack-infra | 08:37 | |
| *** jed56 has joined #openstack-infra | 08:38 | |
| *** dizquierdo has joined #openstack-infra | 08:39 | |
| *** Watanabe has quit IRC | 08:39 | |
| *** comstud has joined #openstack-infra | 08:40 | |
| *** shardy has joined #openstack-infra | 08:40 | |
| *** jlanoux has joined #openstack-infra | 08:41 | |
| *** otter768 has quit IRC | 08:42 | |
| *** unicell1 has joined #openstack-infra | 08:42 | |
| *** gibi has joined #openstack-infra | 08:42 | |
| *** unicell has quit IRC | 08:42 | |
| *** dingyichen has quit IRC | 08:43 | |
| *** ddieterly has joined #openstack-infra | 08:43 | |
| *** mudassirlatif has quit IRC | 08:44 | |
| *** eliqiao has joined #openstack-infra | 08:46 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Add DB API method to get tests from a list of failed runs https://review.openstack.org/268328 | 08:48 |
| *** ddieterly has quit IRC | 08:48 | |
| *** Sukhdev has quit IRC | 08:49 | |
| *** eliqiao_ has quit IRC | 08:49 | |
| *** Jeffrey4l has joined #openstack-infra | 08:49 | |
| *** bauwser is now known as bauzas | 08:54 | |
| *** yfried has quit IRC | 08:55 | |
| *** yfried has joined #openstack-infra | 08:55 | |
| *** yaume has joined #openstack-infra | 08:55 | |
| *** aspiers has joined #openstack-infra | 08:59 | |
| *** fawadkhaliq has quit IRC | 09:01 | |
| *** fawadkhaliq has joined #openstack-infra | 09:01 | |
| *** daemontool has joined #openstack-infra | 09:02 | |
| *** daemontool has quit IRC | 09:03 | |
| *** sflanigan has quit IRC | 09:03 | |
| *** daemontool has joined #openstack-infra | 09:03 | |
| *** daemontool_ has joined #openstack-infra | 09:04 | |
| *** daemontool has quit IRC | 09:08 | |
| *** Daisy has joined #openstack-infra | 09:10 | |
| *** lezbar__ has quit IRC | 09:10 | |
| *** derekh has joined #openstack-infra | 09:11 | |
| *** daemontool_ is now known as daemontool | 09:11 | |
| *** lezbar__ has joined #openstack-infra | 09:11 | |
| *** bharathm has joined #openstack-infra | 09:11 | |
| *** hashar has joined #openstack-infra | 09:13 | |
| *** jistr has joined #openstack-infra | 09:14 | |
| *** HeOS has joined #openstack-infra | 09:17 | |
| *** jaosorior has quit IRC | 09:17 | |
| *** jaosorior has joined #openstack-infra | 09:18 | |
| *** dizquierdo has quit IRC | 09:21 | |
| *** sputnik13 has joined #openstack-infra | 09:21 | |
| *** zeih has quit IRC | 09:22 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Cleanup to use namedtuple https://review.openstack.org/268942 | 09:23 |
| *** Daisy has quit IRC | 09:25 | |
| *** thorst has joined #openstack-infra | 09:26 | |
| *** sputnik13 has quit IRC | 09:26 | |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Only install mock on Python < 3.3 https://review.openstack.org/268945 | 09:27 |
| *** Jeffrey4l has quit IRC | 09:32 | |
| *** thorst has quit IRC | 09:33 | |
| *** mudassirlatif has joined #openstack-infra | 09:35 | |
| *** fhubik is now known as fhubik_brb | 09:39 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Revert "Add CI coverage of deleting the deployed overcloud" https://review.openstack.org/268253 | 09:41 |
| *** ddieterly has joined #openstack-infra | 09:45 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Set DIB_NO_TMPFS=1 https://review.openstack.org/268955 | 09:45 |
| *** rguillebert has joined #openstack-infra | 09:46 | |
| *** ihrachys has joined #openstack-infra | 09:47 | |
| *** sshnaidm has quit IRC | 09:47 | |
| *** ddieterly has quit IRC | 09:49 | |
| *** Jeffrey4l has joined #openstack-infra | 09:50 | |
| *** fhubik_brb is now known as fhubik | 09:56 | |
| openstackgerrit | Merged openstack-infra/groups: Fixed some footer link & added news link https://review.openstack.org/268269 | 09:57 |
| *** sshnaidm has joined #openstack-infra | 10:01 | |
| *** Qiming has quit IRC | 10:04 | |
| *** davideagnello has joined #openstack-infra | 10:05 | |
| *** ociuhandu has quit IRC | 10:06 | |
| *** ociuhandu has joined #openstack-infra | 10:07 | |
| *** ociuhandu has quit IRC | 10:07 | |
| *** pblaho has joined #openstack-infra | 10:07 | |
| *** ociuhandu has joined #openstack-infra | 10:08 | |
| *** ociuhandu has quit IRC | 10:08 | |
| *** davideagnello has quit IRC | 10:10 | |
| *** e0ne has joined #openstack-infra | 10:11 | |
| *** mudassirlatif has quit IRC | 10:11 | |
| *** mudassirlatif has joined #openstack-infra | 10:12 | |
| *** marcusvrn_ has joined #openstack-infra | 10:13 | |
| openstackgerrit | Miles Gould proposed openstack/requirements: Bump minimum python-ironicclient version https://review.openstack.org/266320 | 10:17 |
| *** mika has quit IRC | 10:18 | |
| *** Jeffrey4l has quit IRC | 10:21 | |
| *** Hal has joined #openstack-infra | 10:23 | |
| *** Hal is now known as Guest68594 | 10:23 | |
| *** cdent has joined #openstack-infra | 10:23 | |
| *** mika has joined #openstack-infra | 10:26 | |
| *** mika has joined #openstack-infra | 10:26 | |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Adding support for maven-metadata-plugin https://review.openstack.org/257430 | 10:28 |
| *** ldnunes has joined #openstack-infra | 10:29 | |
| *** thorst has joined #openstack-infra | 10:31 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:33 | |
| *** ociuhandu has joined #openstack-infra | 10:36 | |
| *** thorst has quit IRC | 10:37 | |
| *** otter768 has joined #openstack-infra | 10:38 | |
| *** bradjones has quit IRC | 10:38 | |
| *** bradjones has joined #openstack-infra | 10:40 | |
| *** bradjones has quit IRC | 10:40 | |
| *** bradjones has joined #openstack-infra | 10:40 | |
| *** gildub has quit IRC | 10:43 | |
| *** otter768 has quit IRC | 10:43 | |
| *** ddieterly has joined #openstack-infra | 10:45 | |
| *** ociuhandu has quit IRC | 10:47 | |
| *** ddieterly has quit IRC | 10:51 | |
| esikachev | need merge https://review.openstack.org/#/c/268145/ | 10:52 |
| *** pahuang has quit IRC | 10:53 | |
| *** dims has joined #openstack-infra | 10:54 | |
| *** mudassirlatif has quit IRC | 10:58 | |
| *** sfinucan has joined #openstack-infra | 10:59 | |
| *** lucas-dinner is now known as lucasagomes | 11:09 | |
| *** andrearosa has joined #openstack-infra | 11:09 | |
| *** _nadya_ has joined #openstack-infra | 11:10 | |
| andrearosa | with the new gerrit, when I submit a new PS the votes get on the previous one are still there, is that correct? | 11:11 |
| *** ociuhandu has joined #openstack-infra | 11:11 | |
| *** flaper87 has quit IRC | 11:11 | |
| *** flaper87 has joined #openstack-infra | 11:11 | |
| *** aysyd has joined #openstack-infra | 11:12 | |
| *** amotoki has quit IRC | 11:13 | |
| *** ociuhandu has quit IRC | 11:14 | |
| SotK | Is there anyone around who could paste a copy of /etc/storyboard/storyboard.conf from storyboard.openstack.org? (with private details removed, obviously) | 11:15 |
| *** _et_ has quit IRC | 11:15 | |
| andreaf | andrearosa: I don't think that's true in general, only if you did a rebase with no conflict to resolve | 11:19 |
| andreaf | andrearosa: but a -2 will stick I believe | 11:19 |
| *** claudiub has joined #openstack-infra | 11:19 | |
| andrearosa | andreaf: thanks for the reply, I have an example of one of my patches. I put a new PS with a code change so not only a rebase and I still get the -1 and the +1s I got before | 11:20 |
| andrearosa | andreaf: let me check again and I'll come back to you. thanks | 11:22 |
| *** zhurong has quit IRC | 11:22 | |
| *** ociuhandu has joined #openstack-infra | 11:23 | |
| *** mase_x200 has joined #openstack-infra | 11:24 | |
| *** Qiming has joined #openstack-infra | 11:30 | |
| *** shardy_ has joined #openstack-infra | 11:30 | |
| *** Ramanjaneya has quit IRC | 11:31 | |
| *** shardy has quit IRC | 11:33 | |
| *** kzaitsev_mb has quit IRC | 11:36 | |
| AJaeger | andrearosa: which change? | 11:36 |
| AJaeger | andrearosa: andreaf's explanation is correct, -2 is sticky and rest stays after rebase | 11:37 |
| *** nmagnezi has quit IRC | 11:38 | |
| *** ddieterly has joined #openstack-infra | 11:46 | |
| *** changzhi has quit IRC | 11:47 | |
| jokke_ | review.openstack.org is returning 503s quite a lot today. Anyone aware of this or is it just me seeing it? | 11:48 |
| *** mase_x200 has quit IRC | 11:50 | |
| *** ddieterly has quit IRC | 11:50 | |
| *** oomichi is now known as oomichi_away | 11:51 | |
| andrearosa | AJaeger: I can't show you anymore as I updated new patches and now the old votes are gone, odd, something strange happened. Probably my fault messing up with a chain of dependencies | 11:51 |
| *** yamamoto_ has quit IRC | 11:51 | |
| andrearosa | AJaeger: andreaf thank you both | 11:52 |
| *** nmagnezi has joined #openstack-infra | 11:52 | |
| *** rfolco has joined #openstack-infra | 11:53 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Fix typo https://review.openstack.org/269004 | 11:53 |
| AJaeger | andrearosa: let me check quickly - can you provide me with the URL, please? | 11:53 |
| andrearosa | AJaeger: https://review.openstack.org/259528 the problem was between PS6 and PS7 | 11:54 |
| andrearosa | in the PS7 I addressed some concerns I had on PS6 but all the votes didn't go away | 11:54 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins in /Openstack https://review.openstack.org/269007 | 11:54 |
| *** kzaitsev_mb has joined #openstack-infra | 11:56 | |
| AJaeger | andrearosa: indeed to late to check now. | 11:56 |
| *** rfolco has quit IRC | 11:57 | |
| AJaeger | jokke_: I get the 503s also also ;( | 11:57 |
| andrearosa | AJaeger: np thanks for your time. I'll bug you in case it happens again | 11:58 |
| *** daemontool has quit IRC | 11:58 | |
| *** daemontool has joined #openstack-infra | 11:59 | |
| *** cdent has quit IRC | 12:00 | |
| *** Guest68594 has quit IRC | 12:03 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql: Cleanup to use namedtuple https://review.openstack.org/268942 | 12:04 |
| *** daemontool_ has joined #openstack-infra | 12:04 | |
| *** dizquierdo has joined #openstack-infra | 12:05 | |
| *** daemontool has quit IRC | 12:07 | |
| *** dims has quit IRC | 12:10 | |
| *** fawadkhaliq has quit IRC | 12:13 | |
| *** _nadya_ has quit IRC | 12:14 | |
| *** esikachev has quit IRC | 12:15 | |
| *** wei-wang has quit IRC | 12:17 | |
| *** Hal has joined #openstack-infra | 12:20 | |
| *** Hal is now known as Guest94043 | 12:20 | |
| *** tdurakov is now known as Guest16620 | 12:22 | |
| *** daemontool_ has quit IRC | 12:24 | |
| *** daemontool_ has joined #openstack-infra | 12:25 | |
| *** daemontool_ has quit IRC | 12:25 | |
| *** fhubik is now known as fhubik_brb | 12:25 | |
| *** fhubik_brb is now known as fhubik | 12:25 | |
| *** daemontool_ has joined #openstack-infra | 12:25 | |
| *** ociuhandu_ has joined #openstack-infra | 12:26 | |
| *** ociuhandu has quit IRC | 12:28 | |
| *** ociuhandu_ is now known as ociuhandu | 12:28 | |
| *** baoli has joined #openstack-infra | 12:28 | |
| *** ddieterly has joined #openstack-infra | 12:29 | |
| *** bradjones has quit IRC | 12:30 | |
| samueldmq | I am getting a couple of 503 on gerrit, anyone else able to reproduce the issue? | 12:30 |
| *** daemontool_ has quit IRC | 12:31 | |
| dulek | Sam-I-Am: I'm also experiencing that. | 12:31 |
| openstackgerrit | Merged openstack-infra/project-config: Add sahara tests repository to PROJECTS https://review.openstack.org/268145 | 12:31 |
| samueldmq | dulek: thanks, I think that ^ was directed to me ? | 12:32 |
| odyssey4me | samueldmq dulek I saw more of them last night, but they do seem to show up from time to time. I've not been getting them this morning. | 12:33 |
| odyssey4me | There doesn't appear to be much consistency from what I've seen so far. | 12:33 |
| *** baoli has quit IRC | 12:33 | |
| *** gordc has joined #openstack-infra | 12:33 | |
| *** tdurakov has joined #openstack-infra | 12:33 | |
| dulek | samueldmq: That's right, sorry for pinging you Sam-I-Am! | 12:33 |
| *** fhubik is now known as fhubik_brb | 12:34 | |
| *** cfarquhar has joined #openstack-infra | 12:34 | |
| *** cfarquhar has quit IRC | 12:34 | |
| *** cfarquhar has joined #openstack-infra | 12:34 | |
| *** chlong has joined #openstack-infra | 12:36 | |
| *** electrofelix has joined #openstack-infra | 12:37 | |
| samueldmq | odyssey4me: perhaps CPU hitting 100% from times to times ? | 12:38 |
| *** yamamoto has joined #openstack-infra | 12:38 | |
| *** otter768 has joined #openstack-infra | 12:39 | |
| *** fawadkhaliq has joined #openstack-infra | 12:40 | |
| *** markmcd__ has joined #openstack-infra | 12:42 | |
| electrofelix | hashar waynr zxiiro: looks like AJaeger has responded to the request for feedback on https://review.openstack.org/255252 so looks like merged, would one of you like to complete the approval? | 12:43 |
| *** ashishb has quit IRC | 12:44 | |
| *** otter768 has quit IRC | 12:44 | |
| *** markmcd1 has quit IRC | 12:44 | |
| *** markmcd has quit IRC | 12:44 | |
| *** thorst has joined #openstack-infra | 12:45 | |
| *** markmcd has joined #openstack-infra | 12:46 | |
| hashar | electrofelix: indeed :-) | 12:46 |
| hashar | electrofelix: that will land https://review.openstack.org/#/c/255280/3 "Remove linkcheck from sphinx builders list in config" | 12:47 |
| hashar | though we need openstack infra to update jjjb/zuul as well grrr https://review.openstack.org/#/c/255277/ :D | 12:47 |
| electrofelix | one step at a time ;) | 12:48 |
| hashar | poke me as needed, I tested those last week iirc | 12:48 |
| *** _et_ has joined #openstack-infra | 12:48 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add separate linkcheck env and allow dev to select builder https://review.openstack.org/255252 | 12:49 |
| electrofelix | So if someone from Infra could do the needful on https://review.openstack.org/255277, be much obliged | 12:49 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Set DIB_NO_TMPFS=1 https://review.openstack.org/268955 | 12:50 |
| AJaeger | electrofelix: then let me approve the infra side as well ;) | 12:51 |
| electrofelix | AJaeger: thanks :) | 12:51 |
| *** amotoki has joined #openstack-infra | 12:52 | |
| *** Ramanjaneya has joined #openstack-infra | 12:53 | |
| electrofelix | I'll trigger a recheck on the JJB change https://review.openstack.org/255280 about 30 minutes after that lands to make sure the new job works as expected | 12:53 |
| AJaeger | electrofelix: good idea, thanks | 12:56 |
| openstackgerrit | Merged openstack-infra/project-config: Add separate non-voting job to check doc links for JJB https://review.openstack.org/255277 | 12:57 |
| *** bharathm has quit IRC | 13:00 | |
| *** mase_x200 has joined #openstack-infra | 13:00 | |
| *** rfolco has joined #openstack-infra | 13:01 | |
| *** diltram has joined #openstack-infra | 13:01 | |
| diltram | hello everyone | 13:01 |
| diltram | I have a problem | 13:01 |
| *** rfolco has quit IRC | 13:01 | |
| diltram | I merged my account on launchpad | 13:01 |
| *** rfolco has joined #openstack-infra | 13:02 | |
| diltram | and I lost possibility to login to my account on review.openstack.org | 13:02 |
| diltram | I cannot on "new" account set my username because it's saying that it's used already | 13:02 |
| diltram | may anyone help me with that? | 13:03 |
| diltram | is it possible to merge accounts on review.openstack? | 13:03 |
| *** dprince has joined #openstack-infra | 13:04 | |
| *** doug-fish has joined #openstack-infra | 13:04 | |
| Zara | SotK asked a couple of hours ago: could anyone paste us a copy of /etc/storyboard/storyboard.conf from storyboard.openstack.org? (with private details removed) we need it for debugging email notifications; they work on other instances but not here... | 13:05 |
| *** zz_dimtruck is now known as dimtruck | 13:10 | |
| *** dims has joined #openstack-infra | 13:11 | |
| *** edmondsw has joined #openstack-infra | 13:12 | |
| *** _nadya_ has joined #openstack-infra | 13:14 | |
| *** rcernin has quit IRC | 13:19 | |
| *** _nadya_ has quit IRC | 13:19 | |
| *** dkranz has joined #openstack-infra | 13:20 | |
| *** daemontool has joined #openstack-infra | 13:21 | |
| *** mika has quit IRC | 13:22 | |
| *** zhurong has joined #openstack-infra | 13:22 | |
| *** mika has joined #openstack-infra | 13:23 | |
| *** shardy_ has quit IRC | 13:23 | |
| *** fhubik_brb is now known as fhubik | 13:23 | |
| *** shardy has joined #openstack-infra | 13:24 | |
| *** _nadya_ has joined #openstack-infra | 13:25 | |
| *** moravec has quit IRC | 13:25 | |
| *** trown|outttypeww is now known as trown | 13:29 | |
| *** weshay_xchat has joined #openstack-infra | 13:31 | |
| *** weshay_xchat is now known as weshay | 13:31 | |
| *** jordanP has joined #openstack-infra | 13:32 | |
| *** kzaitsev_mb has quit IRC | 13:33 | |
| *** links has quit IRC | 13:33 | |
| *** fawadkhaliq has quit IRC | 13:34 | |
| *** aojea has quit IRC | 13:34 | |
| *** itsuugo has joined #openstack-infra | 13:35 | |
| *** Ramanjaneya has quit IRC | 13:37 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-jenkins: Customize jenkins.default by template https://review.openstack.org/262787 | 13:38 |
| mordred | Zara: one sec | 13:39 |
| *** peristeri has joined #openstack-infra | 13:41 | |
| mordred | Zara: http://paste.openstack.org/show/484146 | 13:42 |
| *** keedya has quit IRC | 13:42 | |
| Zara | mordred: thank you! :) | 13:42 |
| Zara | SotK ^ | 13:42 |
| SotK | mordred: thanks! | 13:43 |
| *** dimtruck is now known as zz_dimtruck | 13:44 | |
| *** binbincong has quit IRC | 13:45 | |
| *** dingyichen has joined #openstack-infra | 13:45 | |
| *** dtardivel has joined #openstack-infra | 13:45 | |
| *** tlian has joined #openstack-infra | 13:45 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:46 | |
| *** binbincong has joined #openstack-infra | 13:46 | |
| *** zz_dimtruck is now known as dimtruck | 13:47 | |
| odyssey4me | oh dear, gerrit 502's | 13:53 |
| EmilienM | ouch, proxy error | 13:53 |
| mkoderer | yep same for me | 13:53 |
| odyssey4me | I can't say whoops any more :p | 13:53 |
| odyssey4me | 502 gone now | 13:53 |
| *** Liuqing has quit IRC | 13:54 | |
| *** zhurong has quit IRC | 13:55 | |
| *** mika has quit IRC | 13:55 | |
| AJaeger | odyssey4me: these 502s come and go ;( | 13:55 |
| DuncanT | gerrit is very sick it seems | 13:56 |
| *** mika has joined #openstack-infra | 13:57 | |
| *** lennyb_ has quit IRC | 13:57 | |
| *** Ramanjaneya has joined #openstack-infra | 13:57 | |
| *** dimtruck is now known as zz_dimtruck | 13:58 | |
| *** bharathm has joined #openstack-infra | 14:00 | |
| *** edmondsw has quit IRC | 14:02 | |
| *** binbincong has quit IRC | 14:03 | |
| *** dslevin has quit IRC | 14:03 | |
| *** bharathm has quit IRC | 14:05 | |
| *** lennyb has joined #openstack-infra | 14:06 | |
| *** binbincong has joined #openstack-infra | 14:06 | |
| *** regXboi has joined #openstack-infra | 14:07 | |
| *** mase_x200 has quit IRC | 14:12 | |
| *** dkliban has joined #openstack-infra | 14:13 | |
| electrofelix | hashar: gate-jenkins-job-builder-tox-docs-linkcheck is working so we can land https://review.openstack.org/255280 | 14:13 |
| *** rlandy has joined #openstack-infra | 14:14 | |
| *** dtantsur is now known as dtantsur|brb | 14:17 | |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Update to os-brick 0.8.0 https://review.openstack.org/267725 | 14:21 |
| *** keedya has joined #openstack-infra | 14:22 | |
| openstackgerrit | Merged openstack-infra/project-config: tooz: gate on etcd https://review.openstack.org/263678 | 14:23 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Improve usability of the "New Worklist" modal https://review.openstack.org/263728 | 14:25 |
| *** jaypipes has joined #openstack-infra | 14:25 | |
| *** amitgandhinz has joined #openstack-infra | 14:25 | |
| *** yamamoto has quit IRC | 14:25 | |
| *** julim has joined #openstack-infra | 14:26 | |
| *** Ramanjaneya has quit IRC | 14:27 | |
| *** lucasagomes is now known as lucas-hungry | 14:28 | |
| *** sdake has joined #openstack-infra | 14:29 | |
| *** rossella_s has quit IRC | 14:31 | |
| *** rossella_s has joined #openstack-infra | 14:31 | |
| *** daemontool has quit IRC | 14:35 | |
| hashar | electrofelix: nice ! thank you for the link check work :-} | 14:36 |
| hashar | did W+1 | 14:36 |
| *** ddieterly has quit IRC | 14:37 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Remove linkcheck from sphinx builders list in config https://review.openstack.org/255280 | 14:38 |
| *** fawadkhaliq has joined #openstack-infra | 14:38 | |
| *** tsg has joined #openstack-infra | 14:39 | |
| *** otter768 has joined #openstack-infra | 14:40 | |
| *** daemontool has joined #openstack-infra | 14:41 | |
| *** moravec has joined #openstack-infra | 14:42 | |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| * anteaya reads Sam-I-Am's extensive research essay on mtu | 14:43 | |
| *** ddieterly has joined #openstack-infra | 14:44 | |
| pabelanger | morning | 14:44 |
| *** amotoki has quit IRC | 14:44 | |
| *** otter768 has quit IRC | 14:44 | |
| fungi | holiday here so i'm not paying super close attention, though i am around if something urgent needs my help | 14:46 |
| fungi | just give me a heads up | 14:46 |
| *** Jeffrey4l has joined #openstack-infra | 14:47 | |
| anteaya | fungi: will do, happy Martin Luther King Jr day | 14:47 |
| anteaya | morning pabelanger | 14:47 |
| anteaya | pabelanger: hey I just found out about this and I'm thinking of going: https://lists.ubuntu.com/archives/ubuntu-ca/2016-January/011134.html | 14:48 |
| *** ari has joined #openstack-infra | 14:48 | |
| *** bcourt has joined #openstack-infra | 14:48 | |
| *** naggappan has quit IRC | 14:48 | |
| anteaya | pabelanger: any chance you can make it to Kitchener on the 30th? would be great to see you | 14:48 |
| *** pradk has joined #openstack-infra | 14:49 | |
| *** _nadya_ has quit IRC | 14:50 | |
| anteaya | if an infra-root cares to take a peek at java melody and see where we are with gc, that would be great, thank you | 14:51 |
| *** dkliban has left #openstack-infra | 14:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove a done todo list item https://review.openstack.org/269120 | 14:52 |
| *** superdan is now known as dansmith | 14:54 | |
| *** cdent has joined #openstack-infra | 14:54 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:55 | |
| electrofelix | hashar: now we just need to fix the bad links so when sphinx releases with the fix for the broken links we can start to watch out for problems. | 14:56 |
| *** cdent has quit IRC | 14:56 | |
| openstackgerrit | Imre Farkas proposed openstack-infra/project-config: Add openstack/python-wsmanclient project https://review.openstack.org/269122 | 14:57 |
| *** mrmartin has joined #openstack-infra | 14:57 | |
| pabelanger | anteaya: sadly no, already have plans that day. Good news however, I'm all booked up for Austin! | 14:58 |
| anteaya | pabelanger: yay! | 14:58 |
| anteaya | that is good news | 14:58 |
| anteaya | yes the Linux thing is very short notice, hoping to meet some folks and encourage them to announce things a litte further in advance | 14:58 |
| *** fhubik is now known as fhubik_brb | 14:59 | |
| mrmartin | #success I found the reason of openid association handling bug between openstackid.org and askbot. | 14:59 |
| openstackstatus | mrmartin: Added success to Success page | 14:59 |
| * mrmartin is happy now :) | 14:59 | |
| *** keedya_ has joined #openstack-infra | 15:00 | |
| anteaya | pabelanger: Austin is a great city, lots of live music, lots of really interesting buildings | 15:00 |
| *** keedya has quit IRC | 15:00 | |
| anteaya | mrmartin: congratulations | 15:00 |
| mrmartin | thnx :) | 15:00 |
| *** burgerk has joined #openstack-infra | 15:00 | |
| pabelanger | anteaya: Ya, in on Saturday, out friday night ATM | 15:00 |
| mrmartin | 90% dev and debug environment building 10% bug tracing and testing. OMG. | 15:01 |
| pabelanger | anteaya: never been to texas, so should be interesting | 15:01 |
| samueldmq | anteaya: I am looking forward to visiting Austin :) | 15:01 |
| anteaya | pabelanger: texas is interesting, no doubt about it | 15:01 |
| anteaya | samueldmq: oh wonderful, it is a very nice city | 15:01 |
| anteaya | easy to walk around | 15:01 |
| anteaya | the government buildings are impressive | 15:01 |
| pabelanger | The stars at night, are big and bright! | 15:02 |
| anteaya | pabelanger: ha ha ha | 15:02 |
| pabelanger | anteaya: only thing I know about Texas: https://www.youtube.com/watch?v=ZJEwrw4VEls | 15:02 |
| samueldmq | anteaya: I am coming to keystone midcycle later this month, then I will visit it before summit :) | 15:03 |
| samueldmq | anteaya: and then at the summit again, woot | 15:03 |
| *** _nadya_ has joined #openstack-infra | 15:03 | |
| hashar | electrofelix: excellent :-} | 15:03 |
| anteaya | samueldmq: yay twice | 15:03 |
| anteaya | pabelanger: will watch in a minute | 15:04 |
| samueldmq | :) | 15:04 |
| *** zz_dimtruck is now known as dimtruck | 15:04 | |
| *** mriedem has joined #openstack-infra | 15:05 | |
| *** Jeffrey4l has quit IRC | 15:08 | |
| *** Qiming has quit IRC | 15:09 | |
| *** lezbar__ has quit IRC | 15:10 | |
| *** dingyichen has quit IRC | 15:11 | |
| *** Jeffrey4l has joined #openstack-infra | 15:11 | |
| *** fhubik_brb is now known as fhubik | 15:12 | |
| *** lezbar has joined #openstack-infra | 15:14 | |
| *** tdurakov has quit IRC | 15:14 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins https://review.openstack.org/269007 | 15:17 |
| *** tdurakov has joined #openstack-infra | 15:17 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/puppet-log_processor: Allow for python deps to be already defined https://review.openstack.org/269136 | 15:19 |
| *** baoli has joined #openstack-infra | 15:20 | |
| andreaf | yolanda, clarkb: follow-up to https://review.openstack.org/#/c/266982/ - other libraries that may generate a conflict in puppet log_processor | 15:20 |
| *** tdurakov has quit IRC | 15:21 | |
| *** Jeffrey4l has quit IRC | 15:21 | |
| openstackgerrit | Imre Farkas proposed openstack-infra/project-config: Add openstack/python-wsmanclient project https://review.openstack.org/269122 | 15:24 |
| *** baoli has quit IRC | 15:25 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Add python-jobs and python3-jobs to freezer* https://review.openstack.org/269138 | 15:26 |
| *** baoli has joined #openstack-infra | 15:26 | |
| *** lucas-hungry is now known as lucasagomes | 15:28 | |
| *** med_ has quit IRC | 15:28 | |
| anteaya | pabelanger: nice | 15:29 |
| anteaya | pabelanger: I wonder if you feel like trying that | 15:29 |
| *** baoli has quit IRC | 15:29 | |
| *** mrmartin has quit IRC | 15:30 | |
| anteaya | I think it is time for some breakfast and back to the mtu research mailing list post | 15:30 |
| *** chlong has quit IRC | 15:31 | |
| *** HeOS has quit IRC | 15:31 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Adjust meeting time for US/EU doc-install-team-meeting https://review.openstack.org/266987 | 15:34 |
| *** amitgandhinz has quit IRC | 15:34 | |
| *** baoli has joined #openstack-infra | 15:34 | |
| *** amitgandhinz has joined #openstack-infra | 15:35 | |
| *** _nadya_ has quit IRC | 15:35 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins https://review.openstack.org/269007 | 15:38 |
| *** lezbar has quit IRC | 15:38 | |
| *** Sukhdev has joined #openstack-infra | 15:39 | |
| *** lezbar has joined #openstack-infra | 15:39 | |
| *** mtanino has joined #openstack-infra | 15:40 | |
| *** annegentle has joined #openstack-infra | 15:40 | |
| *** baoli has quit IRC | 15:42 | |
| *** bharathm has joined #openstack-infra | 15:42 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Move Tuskar to openstack-attic https://review.openstack.org/269149 | 15:42 |
| *** palendae has joined #openstack-infra | 15:43 | |
| *** tsg has quit IRC | 15:44 | |
| *** sdake_ has joined #openstack-infra | 15:44 | |
| palendae | Is there an IP range for the infra's git servers? If so, I'd like to submit them to my work network for whitelisting | 15:45 |
| anteaya | infra-root ^^ | 15:45 |
| anteaya | palendae: I don't know the answer to that question but stick around so that someone who knows can reply | 15:46 |
| *** sabeen1 has joined #openstack-infra | 15:46 | |
| anteaya | also it is a holiday in the US today so it might be a while before you get a reply | 15:46 |
| *** bharathm has quit IRC | 15:47 | |
| mordred | palendae: not really, no, sorry | 15:47 |
| palendae | anteaya: Yep, no worries | 15:47 |
| *** sdake has quit IRC | 15:48 | |
| mordred | they're cloud servers at rackspace - so it's essentially random IPs from the rackspace pool | 15:48 |
| *** sabeen3 has joined #openstack-infra | 15:48 | |
| palendae | mordred: For doing git-review pushes? | 15:48 |
| mordred | palendae: git review pushes go to the gerrit server - and that is a single IP that we announce in advance if we need to move it | 15:49 |
| mordred | 104.130.159.134 | 2001:4800:7818:102:be76:4eff:fe05:9b12 | 15:50 |
| palendae | mordred: Ah, gotcha | 15:50 |
| palendae | mordred: Thanks! | 15:50 |
| mordred | sure thing! | 15:50 |
| *** annegentle has quit IRC | 15:50 | |
| anteaya | palendae: ah sorry I mis-understood | 15:50 |
| anteaya | mordred: thank you | 15:50 |
| palendae | I worded it poorly :p | 15:50 |
| anteaya | glad you got what you needed | 15:50 |
| *** sabeen1 has quit IRC | 15:51 | |
| *** bcourt has quit IRC | 15:51 | |
| anteaya | palendae: you might like this: http://docs.openstack.org/infra/manual/developers.html#quick-reference | 15:51 |
| anteaya | a graphic representation of our workflow | 15:51 |
| *** mpavone has quit IRC | 15:52 | |
| anteaya | atop the rest of our developer user manual | 15:52 |
| *** hdd has joined #openstack-infra | 15:52 | |
| *** resker has quit IRC | 15:53 | |
| *** kgiusti has joined #openstack-infra | 15:54 | |
| *** abregman has quit IRC | 15:56 | |
| *** aopincar has quit IRC | 15:57 | |
| *** baoli has joined #openstack-infra | 15:57 | |
| anteaya | Sam-I-Am: going over your mtc research, now I have never looked at logs to evaluate for dropped packets due to the fact mtu was exceeded, so when you get a change I would be grateful for a tour of https://gist.github.com/ionosphere80/9eb0e2c0b3e780de9afc so I can learn what I should be looking for, here or -neutron or -dev, whereever is best | 15:58 |
| anteaya | s/mtc/mtu | 15:58 |
| anteaya | s/change/chance | 15:58 |
| *** rhallisey has joined #openstack-infra | 15:59 | |
| *** angdraug has joined #openstack-infra | 15:59 | |
| *** links has joined #openstack-infra | 16:00 | |
| *** sdake_ is now known as sdake | 16:01 | |
| *** isaacb has quit IRC | 16:02 | |
| *** dtantsur|brb is now known as dtantsur | 16:02 | |
| *** keedya has joined #openstack-infra | 16:02 | |
| *** bcourt has joined #openstack-infra | 16:03 | |
| *** yfried has quit IRC | 16:03 | |
| *** baoli has quit IRC | 16:04 | |
| *** flip214 has quit IRC | 16:09 | |
| clarkb | palendae: anteaya mordred also you can push over https and use your http proxy as ana lternative | 16:10 |
| *** erlon has joined #openstack-infra | 16:10 | |
| *** baoli has joined #openstack-infra | 16:10 | |
| clarkb | this may be more desireable in some environments | 16:10 |
| *** armax has joined #openstack-infra | 16:12 | |
| anteaya | but still pushing to gerrit not to git.openstack.org | 16:12 |
| clarkb | yes | 16:12 |
| gtmanfred | mordred: Shrews yall get a chance to check this https://review.openstack.org/#/c/268404 ? | 16:12 |
| anteaya | clarkb: thanks I always forget about the REST api | 16:12 |
| *** vivekd has joined #openstack-infra | 16:12 | |
| anteaya | clarkb: how was the sportsball yesterday? | 16:13 |
| gtmanfred | sportsball was less awesome yesterday, AMAZING ON SATURDAY | 16:13 |
| palendae | clarkb: Is that just as simple as making sure that you have an https remote set for pushing in your git checkout? | 16:13 |
| clarkb | the sea chickens were still asleep on the field and the large cats nommed on them | 16:13 |
| clarkb | palendae: and using your gerrit http password yes | 16:14 |
| gtmanfred | until the second half when they woke up, but were missing one wing | 16:14 |
| *** baoli has quit IRC | 16:14 | |
| anteaya | clarkb: :( | 16:14 |
| *** annegentle has joined #openstack-infra | 16:15 | |
| anteaya | clarkb: enjoy your holiday | 16:15 |
| *** edmondsw has joined #openstack-infra | 16:16 | |
| *** zeih has joined #openstack-infra | 16:17 | |
| openstackgerrit | Merged openstack-infra/release-tools: pick a different previous version if the current version exists https://review.openstack.org/267684 | 16:18 |
| *** eharney has joined #openstack-infra | 16:20 | |
| fungi | on the gc activity question, here's the weekly graph (so you can compare to last monday and see where it went before we finally restarted gerrit) http://fungi.yuggoth.org/tmp/gc.png | 16:20 |
| mriedem | is it just me or is logstash.o.o not loading? | 16:20 |
| fungi | i have a feeling we may want to go ahead and restart gerrit now rather than wait for it to get worse later today | 16:21 |
| *** Guest94043 has quit IRC | 16:21 | |
| fungi | infra-root: ^ | 16:21 |
| mordred | fungi: ++ | 16:21 |
| fungi | i'm willing to do that in a few minutes | 16:21 |
| *** flip214 has joined #openstack-infra | 16:22 | |
| *** flip214 has quit IRC | 16:22 | |
| *** flip214 has joined #openstack-infra | 16:22 | |
| openstackgerrit | Merged openstack-infra/release-tools: remove use of virtualenv from release scripts https://review.openstack.org/268308 | 16:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Just let shade and os-client-config handle clouds https://review.openstack.org/226758 | 16:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 16:22 |
| mordred | Shrews: ^^ I have rebased the nodepool consume-shade patches | 16:23 |
| *** ddieterly has quit IRC | 16:23 | |
| *** links has quit IRC | 16:23 | |
| fungi | the open files count steadily grew until it reached ~5k on saturday, then was abruptly knocked down to under 1k around midnight utc heading into sunday. i guess we have some timed cleanup happening there? | 16:24 |
| anteaya | fungi: thank you, is there any way I can generate that kind of graph for myself? | 16:25 |
| anteaya | mriedem: I confirm, logstash.o.o doens't give many anything beyond a black background | 16:26 |
| fungi | anteaya: no, the javamelody gerrit plugin is restricted to administrators because it has destructive options, it's not a read-only interface. though someone was looking at a way to make these accessible via a net-snmp oid or something i think | 16:26 |
| anteaya | s/many/me | 16:26 |
| *** isaacb has joined #openstack-infra | 16:26 | |
| anteaya | fungi: okay, thanks for generating that graph | 16:26 |
| fungi | in which case we could add them to cacti | 16:26 |
| *** woodster_ has joined #openstack-infra | 16:27 | |
| anteaya | oh cacti would be awesome, alas I don't know how to do that work :( | 16:27 |
| *** sdake has quit IRC | 16:27 | |
| *** tonytan4ever has joined #openstack-infra | 16:27 | |
| *** hashar has quit IRC | 16:28 | |
| stevemar | anyone have a few minutes to chat about: https://review.openstack.org/#/c/268826/ ? | 16:28 |
| fungi | also http errors tracked in javamelody are way up starting around 1500 utc friday | 16:28 |
| *** bnemec has quit IRC | 16:29 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl. https://review.openstack.org/269174 | 16:29 |
| anteaya | stevemar: it seems the sticking point is the increase in timeout for the job | 16:29 |
| anteaya | and I agree with AJaeger 3 hours for a timeout is far too much for a job in the integrated gate | 16:30 |
| stevemar | anteaya: since around jan 15th, the gate-tempest-dsvm-keystone-eventlet-full seems to be hitting the 2 hr timeout very easily | 16:30 |
| anteaya | stevemar: yes, that alone would be a cause for concern in my book | 16:30 |
| fungi | stevemar: any idea what changed a few days ago to make that job so much slower? | 16:31 |
| stevemar | anteaya: you can check the amount of times we had to recheck here: https://review.openstack.org/#/c/264068/ | 16:31 |
| stevemar | fungi: no idea | 16:31 |
| anteaya | stevemar: no I believe you | 16:31 |
| anteaya | stevemar: yeah I think finding out why the job is slower is the route I would encourage | 16:31 |
| anteaya | I don't think rechecking is the solution here | 16:31 |
| *** sdake has joined #openstack-infra | 16:31 | |
| stevemar | fungi: anteaya, we created the job a few weeks job, and it's been running fine | 16:31 |
| stevemar | fungi: anteaya sometimes it passes at around 1h 30m to 1h 40m. | 16:32 |
| fungi | stevemar: is it spending more time in tempest than it was, or more time outside tempest? any correlation of timeouts to specific nodepool providers? | 16:32 |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Remove glance python34 jobs on Liberty https://review.openstack.org/269178 | 16:32 |
| anteaya | I'd encourage you to move it from voting to non-voting in check and do further work investigating the job | 16:32 |
| clarkb | in thr past we did have to tune the eventlet procssses | 16:33 |
| stevemar | anteaya: i can do that | 16:33 |
| stevemar | fungi: i was gonna look that up last night :) | 16:33 |
| clarkb | iirc the default of 1 isnt enough and everything slows down when keystone is slow | 16:33 |
| fungi | though i guess it's hard to check that with kibana not loading | 16:33 |
| anteaya | stevemar: yeah, we aren't fans of just increasing timeouts as it usually just masks a bigger problem that you have to investigate anyway | 16:33 |
| stevemar | fungi: yep! that made it extra hard last night to get proof it was timing out :) | 16:33 |
| clarkb | as for kibana not loading that probably means elasticsearch is unhappy | 16:34 |
| stevemar | anteaya: yeah, makes sense. i just wanted something to change since we have 5 or so patches that are trying to gate and days go by without merges | 16:34 |
| anteaya | clarkb: did you upgrade the workers over the weekend or no? | 16:34 |
| *** jaosorior has quit IRC | 16:34 | |
| anteaya | stevemar: yup, I support the change in that job to non-voting in check (removing it from the gate queue) and fixing the job | 16:35 |
| *** jaosorior has joined #openstack-infra | 16:35 | |
| clarkb | anteaya I did not | 16:35 |
| anteaya | stevemar: as clarkb says perhaps eventlet needs tuning | 16:35 |
| anteaya | clarkb: okey dokey | 16:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove a done todo list item https://review.openstack.org/269120 | 16:35 |
| Odd_Bloke | So we're specifying an email publisher as a default in our JJB configuration. However, this doesn't get applied if there are any other publishers (which is, obviously, pretty common). Is there any way that I can get JJB to merge the defaults with the options specified in a job(-template)? | 16:36 |
| fungi | zaro: i'm seeing quite exceptions like these more or less continuously in our gerrit error log at the moment: http://paste.openstack.org/show/484172/ (with various channel numbers) | 16:36 |
| fungi | s/quite// | 16:36 |
| anteaya | the zuul job queue graph is a very steady increase, I don't think I have seen as smooth an incline on that graph before | 16:37 |
| clarkb | Odd_Bloke: no | 16:37 |
| *** med_ has joined #openstack-infra | 16:37 | |
| *** med_ is now known as Guest51217 | 16:37 | |
| *** bnemec has joined #openstack-infra | 16:37 | |
| stevemar | fungi: the jobs on hpcloud seem to be hitting the timeout consistently, and the passes seem to be on rax servers | 16:38 |
| fungi | zaro: aha, looks like maybe that's just poor handling of dropped ssh connections https://issues.apache.org/jira/browse/SSHD-535 | 16:38 |
| stevemar | there are also ovh hosts? /me shrugs | 16:38 |
| anteaya | stevemar: yes | 16:39 |
| anteaya | stevemar: and internap and bluebox | 16:39 |
| fungi | stevemar: that's excellent analysis. aside from the hardware backing those, one of the biggest differences between them is that the images in hpcloud are built with dib while those in rax are built as modified snapshots of provider's base images | 16:39 |
| *** _nadya_ has joined #openstack-infra | 16:39 | |
| anteaya | stevemar: as hpcloud is going away end of next week | 16:39 |
| anteaya | well and we are grateful to new hosts anytime, not just when one host goes away | 16:40 |
| fungi | ovh, bluebox and internap are also using dib, so if you find timeouts on those as well (there are fewer of them so they may not be as easy to find examples for) then that would be a useful data point | 16:40 |
| stevemar | fungi: looks like there were two runs on ovh: one passed, one failed | 16:40 |
| anteaya | and as far as I know, dib everywhere is the way of the future | 16:40 |
| stevemar | err, timed out, whatever | 16:40 |
| *** otter768 has joined #openstack-infra | 16:41 | |
| fungi | did the passing one complete close to the timeout threshold? | 16:41 |
| fungi | are some runs in hpcloud passing too, or are they consistently failing in hpcloud? | 16:41 |
| fungi | wow, our zuul job queue graph is pretty wacky | 16:43 |
| *** fbo_ has quit IRC | 16:44 | |
| fungi | steadily accumulating up to 25k waiting and then a couple hours ago it dropped like a stone. did someone restart something? | 16:44 |
| clarkb | it may be nodepool | 16:44 |
| clarkb | since they share s gearman server now | 16:45 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full https://review.openstack.org/268826 | 16:45 |
| *** fbo has joined #openstack-infra | 16:45 | |
| stevemar | anteaya: fungi ^ as a bandaid, for now | 16:45 |
| *** otter768 has quit IRC | 16:45 | |
| stevemar | when logstash is back up, it'll be easier for me to figure out what's going on | 16:45 |
| fungi | clarkb: the "last reconfigured" time reported by the zuul status page seems to be ~ the same as the sheer dropoff in waiting jobs | 16:46 |
| notmorgan | stevemar: porbably not enough workers. | 16:46 |
| *** jaosorior has quit IRC | 16:46 | |
| notmorgan | stevemar: when we ran 1-worker eventlet originally in the pgsql run we ran into similar extended timeouts | 16:46 |
| *** jaosorior has joined #openstack-infra | 16:47 | |
| notmorgan | everything was hung up on keystone being slow. | 16:47 |
| stevemar | notmorgan: i'm pretty sure bknudson_ just did a rename of the psql job | 16:47 |
| fungi | #status notice Gerrit is restarting quickly as a workaround for performance degradation | 16:47 |
| openstackstatus | fungi: sending notice | 16:47 |
| *** zhenguo has joined #openstack-infra | 16:48 | |
| bknudson_ | might as well check the values for how many threads it's setting. | 16:48 |
| *** jpr1 has joined #openstack-infra | 16:48 | |
| fungi | gerrit _should_ be back up now | 16:49 |
| anteaya | stevemar: I'll peek when the gerrit is back | 16:49 |
| notmorgan | ah | 16:49 |
| -openstackstatus- NOTICE: Gerrit is restarting quickly as a workaround for performance degradation | 16:49 | |
| anteaya | yay | 16:49 |
| stevemar | yay | 16:49 |
| fungi | seems to be working for me | 16:49 |
| *** ddieterly has joined #openstack-infra | 16:50 | |
| openstackstatus | fungi: finished sending notice | 16:50 |
| fungi | clarkb: doesn't look like either zuul or nodepool were restarted a couple hours ago | 16:50 |
| anteaya | stevemar: can you remove that job from the gate queue for keystone please? | 16:51 |
| fungi | clarkb: so i'm starting to wonder whether we recently grew an accounting error in the stats reporting which resets on configuration reload maybe? | 16:51 |
| anteaya | stevemar: non-voing jobs are in check only | 16:51 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full https://review.openstack.org/268826 | 16:52 |
| stevemar | fungi: anteaya done, i always forget that rule.. | 16:52 |
| * anteaya is here to remind you | 16:52 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Optimize core list retrieval in reviewers https://review.openstack.org/268799 | 16:52 |
| * notmorgan votes to just remove the eventlet job totally | 16:52 | |
| anteaya | stevemar: I'm happy thank you | 16:53 |
| notmorgan | and finish the uwsgi thing instead of trying to fix "eventlet being slow" | 16:53 |
| stevemar | notmorgan: looking at two failures now and the evenlet workers is set to 2 | 16:53 |
| anteaya | notmorgan: get the rest of keystone to agree and I can support that patch | 16:53 |
| clarkb | fungi maybe? | 16:53 |
| notmorgan | yeah might need to push it to 4. | 16:53 |
| notmorgan | or... just delete the job and not test eventlet :P | 16:53 |
| notmorgan | for mitaka | 16:53 |
| openstackgerrit | Merged openstack-infra/reviewstats: Add python-brick-cinderclient-ext project to Cinder https://review.openstack.org/268793 | 16:54 |
| stevemar | anteaya: fwiw, eventlet was targeted to be removed in M, but there was some pushback | 16:54 |
| fungi | stevemar: you're also skipping that job on stable branches but didn't explain why in the commit message. i'm okay with that, just making sure it's intentional | 16:54 |
| *** julim has quit IRC | 16:54 | |
| notmorgan | eventlet is soooooo bad. | 16:54 |
| anteaya | fungi: I took it as due to the job just being added a few weeks ago, but I agree with you, the commit message should say so | 16:55 |
| fungi | <imagine fist-shake emoji here> | 16:55 |
| openstackgerrit | Sonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support https://review.openstack.org/268795 | 16:55 |
| notmorgan | fungi: exactly | 16:55 |
| fungi | anteaya: stevemar: yep, that makes sense. if it's intentional and not a copy-paste accident, i'll approve as is | 16:56 |
| stevemar | fungi: hmm, we should still test it with stable, *grumble | 16:56 |
| stevemar | * | 16:56 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full https://review.openstack.org/268826 | 16:57 |
| stevemar | thanks for keeping me honest, i initially thought we just need it on master, but it's applicable to stable, so keep it there | 16:57 |
| *** julim has joined #openstack-infra | 16:57 | |
| stevemar | bknudson_: ^ | 16:57 |
| fungi | stevemar: if the job name is the same on stable branches as master (seems to be) then you can't make it non-voting on just some branches. you could have two versions of the job: a voting on on stable branches and a non-voting one on master. or you could switch to just running the job on stable branches and drop it from master entirely as notmorgan sort of suggested rather than making it non-voting | 16:58 |
| anteaya | stevemar: welcome, thanks for wanting to be honest | 16:58 |
| bknudson_ | stevemar: looks like we're running with 2 admin_workers and public_workers = None | 16:58 |
| stevemar | bknudson_: yeah, shouldn't it default to # of cores? | 16:58 |
| stevemar | or something like that | 16:58 |
| bknudson_ | stevemar: I believe the default is the # of cores. | 16:58 |
| stevemar | None cores, yay | 16:59 |
| stevemar | maybe it meant Nine :P | 16:59 |
| fungi | or nein | 16:59 |
| bknudson_ | you can see the childs starting in the log... | 16:59 |
| mordred | stevemar, notmorgan: I do not understand the keep-eventlet pov even a little bit | 16:59 |
| bknudson_ | "Starting 8 workers" | 16:59 |
| stevemar | mordred: something about containerizing stuff and things, you saw the ML thread | 17:00 |
| bknudson_ | I'd also be happy to drop eventlet. They can run with uwsgi or whatever. | 17:00 |
| stevemar | bknudson_: is that something we had set before? or just used the default? | 17:01 |
| bknudson_ | There's an assumption that the public interface is the one that's used in the tests, but I'm not sure that's the case. | 17:01 |
| mordred | or run a container management service thing that spins up and tears down single-threaded non-eventlet backend container workers | 17:01 |
| mordred | I mean, I hear that's all the rage in containerization land | 17:01 |
| bknudson_ | The comments in devstack says that it's supposed to use the # of cpus. These haven't changed for some time. | 17:01 |
| *** burgerk has quit IRC | 17:02 | |
| stevemar | "oslo_service.service [-] Starting 8 workers" | 17:02 |
| mordred | stevemar: I may have missed that thread | 17:02 |
| *** matrohon has quit IRC | 17:04 | |
| stevemar | mordred: http://lists.openstack.org/pipermail/openstack-operators/2015-November/009019.html and later http://lists.openstack.org/pipermail/openstack-dev/2015-December/080892.html | 17:05 |
| *** dims_ has joined #openstack-infra | 17:05 | |
| notmorgan | i saw that thread | 17:05 |
| notmorgan | and i would still rm -rf eventlet | 17:05 |
| notmorgan | and say "make your container run apache in forground mode or uwsgi or anything else that does the same thing" | 17:06 |
| bknudson_ | they can write their own eventlet wrapper if they want | 17:07 |
| notmorgan | bknudson_: ++ | 17:07 |
| notmorgan | it's the "Dev" part of "DevOps" :P | 17:07 |
| *** isaacb has quit IRC | 17:07 | |
| stevemar | bknudson_: even the runs that pass set public_workers to None: http://logs.openstack.org/41/215041/15/check/gate-tempest-dsvm-keystone-eventlet-full/48dda85/logs/screen-key.txt.gz | 17:08 |
| mordred | k. to be fair, it doens't look like anybody had an answer for the multiple virtualenv question | 17:08 |
| bknudson_ | stevemar: it looks like we're just seeing that some runs are slower than others... maybe because of some network issue or disk issue or whatever | 17:08 |
| *** vivekd has quit IRC | 17:08 | |
| *** dims has quit IRC | 17:08 | |
| stevemar | bknudson_: yeah, that's why i wanted to increase the timeout | 17:09 |
| notmorgan | mordred: aache can handle multiple virtualenvs in mod_wsgi now, it's part of the mod_wsgi line - also mod_uwsgi [uwsgi does it fine] | 17:09 |
| bknudson_ | stevemar: maybe admin_workers=2 is too low? | 17:09 |
| stevemar | bknudson_: seems fine for this rax run: http://logs.openstack.org/41/215041/15/check/gate-tempest-dsvm-keystone-eventlet-full/8f0ca10/logs/screen-key.txt.gz | 17:09 |
| greghaynes | Mod proxy uwsgi will 'just work' with multiple venvs | 17:09 |
| *** armax has quit IRC | 17:09 | |
| mordred | notmorgan: ah - awesome | 17:10 |
| *** davideagnello has joined #openstack-infra | 17:10 | |
| mordred | notmorgan: that's the only thing I could see from the threads that was a legit 'oh, that's a good point' complaint | 17:10 |
| greghaynes | Its possible there is some wierd requirement for not using the proxy though... | 17:10 |
| *** armax has joined #openstack-infra | 17:10 | |
| mordred | complaining that apache is the default when there are other choices people _could_ use is bikeshedding | 17:10 |
| notmorgan | mordred: WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/home/keystone/VENV/lib/python2.7/site-packages [for example] | 17:10 |
| *** jlanoux has quit IRC | 17:10 | |
| *** hockeynut_afk has joined #openstack-infra | 17:10 | |
| mordred | notmorgan: and then you do a different virtualenv for horizon in that apache I guess? | 17:10 |
| notmorgan | mordred: i didn't say it was "good" but that at least makes the DaemonProcess use a specific site-packages | 17:10 |
| notmorgan | mordred: yeah you can do that. | 17:11 |
| mordred | cool. wfm | 17:11 |
| notmorgan | mordred: in-fact i have done that | 17:11 |
| notmorgan | mordred: but you must use daemon mod | 17:11 |
| mordred | notmorgan: neat! | 17:11 |
| notmorgan | mode* | 17:11 |
| notmorgan | not the other mod_wsgi moe | 17:11 |
| notmorgan | mode* | 17:11 |
| *** bgmccollum has left #openstack-infra | 17:11 | |
| notmorgan | ideally you'd use uwsgi as the daemon because uwsgi natively handles venvs | 17:11 |
| mordred | yah. that sounds like the best way forward | 17:12 |
| notmorgan | it also means you don't need to restart apache to reload <service> | 17:12 |
| mordred | and also, folks who prefer nginx - it works great with uwsgi | 17:12 |
| notmorgan | ayup | 17:12 |
| notmorgan | either as raw proxy or uwsgi-native | 17:12 |
| greghaynes | notmorgan: you mean mod_proxy_uwsgi vs mod_uwsgi, right? | 17:12 |
| bknudson_ | do you think it would be better for devstack to reference architecture with uwsgi ? | 17:12 |
| notmorgan | greghaynes: both. | 17:12 |
| notmorgan | bknudson_: no | 17:13 |
| notmorgan | bknudson_: well uwsgi behind apache | 17:13 |
| notmorgan | ;) | 17:13 |
| bknudson_ | right, uwsgi behind apache | 17:13 |
| notmorgan | and i'd run horizon in a separate uwsgi | 17:13 |
| * notmorgan might go write some devstack today. | 17:13 | |
| *** fhubik is now known as fhubik_brb | 17:13 | |
| bknudson_ | devstack already has some SSL proxy stuff in it | 17:13 |
| bknudson_ | so using proxy all the time would make devstack simpler | 17:13 |
| *** mrmartin has joined #openstack-infra | 17:13 | |
| notmorgan | greghaynes: or whatever is the right answer. i think it depends on the apache version | 17:14 |
| mordred | bknudson_: ++ | 17:14 |
| notmorgan | greghaynes: could also just use mod_proxy or mod_rewrite_proxy | 17:14 |
| mordred | of course,I think devstack should run in notmorgan sub-url mode myself :) | 17:14 |
| notmorgan | mordred: working on it! | 17:14 |
| *** fhubik_brb is now known as fhubik | 17:14 | |
| notmorgan | mordred: ;) | 17:14 |
| greghaynes | Ya, last I checked mod_uwsgi was not really advised | 17:14 |
| mordred | notmorgan: can we call that deployment "notmorgan-style" ? | 17:14 |
| bknudson_ | mordred: I've been trying to fix tempest-lib so that it works with keystone on sub-url. | 17:14 |
| *** fhubik has quit IRC | 17:14 | |
| mordred | bknudson_: neat! | 17:15 |
| greghaynes | But if it works too then awesome | 17:15 |
| notmorgan | greghaynes: yeah i also want to get uwsgi to use domain sockets | 17:15 |
| notmorgan | greghaynes: rather than needing to run an "internal" port | 17:15 |
| bknudson_ | the uwsgi proxy setup and subpaths are not incompatible. | 17:15 |
| notmorgan | bknudson_: tempestlib was the blocker for suburl. | 17:15 |
| bknudson_ | and doing always-proxy would make it easier to do in devstack | 17:16 |
| bknudson_ | should say "would make *both* easier to do in devstack" | 17:16 |
| notmorgan | bknudson_: but we'd need to L7 route still to do suburl so we'd have to add HAProxy or nginx in | 17:16 |
| *** jswarren__ has quit IRC | 17:16 | |
| stevemar | fungi: i made the job non-voting on all branches, that's alright i assume? | 17:16 |
| bknudson_ | HAProxy might be the better option over apache. I don't know | 17:17 |
| notmorgan | bknudson_: you'd still need an http service for horizon and keystone | 17:17 |
| notmorgan | bknudson_: and apache is the best of the options since we have better federation support. | 17:17 |
| notmorgan | though we could just run uwsgi in http mode | 17:17 |
| fungi | stevemar: lgtm. approved | 17:17 |
| stevemar | notmorgan, bknudson_ #-keystone? :) | 17:17 |
| bknudson_ | you can run uwsgi to serve http. why convert http->uwsgi->http? Is it just because HTTP 1.0 closed connections? | 17:18 |
| stevemar | fungi: ty sir | 17:18 |
| notmorgan | bknudson_: standard testing for apache-isms since we recommend it. | 17:18 |
| *** hdd has quit IRC | 17:18 | |
| notmorgan | mordred: i mean, i can try and run up a devstack using haproxy and sub-url. i think it'll all work. now. | 17:19 |
| openstackgerrit | Doug Wiegley proposed openstack/requirements: Tweak requirements and directions for constraints scripts https://review.openstack.org/268332 | 17:19 |
| *** e0ne has quit IRC | 17:19 | |
| *** ihrachys has quit IRC | 17:20 | |
| *** dizquierdo has quit IRC | 17:21 | |
| mgagne | is there a page where I can find history and average time of gate-tempest-dsvm-neutron-full job? | 17:24 |
| anteaya | mgagne: http://status.openstack.org/openstack-health/#/g/build_name/gate-tempest-dsvm-neutron-full | 17:26 |
| *** apoorvad has joined #openstack-infra | 17:26 | |
| *** _nadya_ has quit IRC | 17:26 | |
| *** vivekd has joined #openstack-infra | 17:27 | |
| *** _et_ has quit IRC | 17:27 | |
| mgagne | anteaya how can I get average run time? | 17:27 |
| anteaya | mgagne: well that functionality might not be built yet | 17:27 |
| anteaya | it is a fairly new tool | 17:27 |
| anteaya | it might not have all you need | 17:27 |
| mgagne | yea, that's what I'm looking for. I get stats for my provider but would like to know global average to get an idea about trend | 17:28 |
| anteaya | yup makes sense | 17:28 |
| anteaya | mtreinish is the main developer, I'm sure he would be interested in hearing your thoughts | 17:28 |
| anteaya | here is the repo: http://git.openstack.org/cgit/openstack/openstack-health/tree/ | 17:29 |
| *** _et_ has joined #openstack-infra | 17:29 | |
| anteaya | and the open patches: https://review.openstack.org/#/q/project:openstack/openstack-health+status:open | 17:30 |
| *** daemontool has quit IRC | 17:30 | |
| *** achanda has joined #openstack-infra | 17:30 | |
| *** daemontool has joined #openstack-infra | 17:31 | |
| jeblair | http://graphite.openstack.org/render/?width=586&height=308&_salt=1453138341.392&target=scale%28averageSeries%28stats.timers.nodepool.job.gate-tempest-dsvm-neutron-full.master.devstack-trusty.*.runtime.mean%29%2C0.00001666666666666666%29. | 17:32 |
| jeblair | mgagne: you may also be interested in playing with graphite a bit ^ | 17:32 |
| *** markus_z has quit IRC | 17:32 | |
| jeblair | mgagne: that's an average of the runtime across all providers | 17:32 |
| jeblair | mgagne: http://graphite.openstack.org/composer/?width=586&height=308&_salt=1453138341.392&target=scale%28averageSeries%28stats.timers.nodepool.job.gate-tempest-dsvm-neutron-full.master.devstack-trusty.*.runtime.mean%29%2C0.00001666666666666666%29. | 17:32 |
| jeblair | mgagne: open it up in the 'composer' url and you can easily edit it | 17:33 |
| *** trown is now known as trown|lunch | 17:33 | |
| mgagne | jeblair trying to play with grafana =) | 17:33 |
| jeblair | mgagne: it's the same data | 17:34 |
| jeblair | mgagne: it's a little easier to do ad-hoc things in graphite's ui though | 17:34 |
| mgagne | jeblair I'm a fancy guy | 17:34 |
| jeblair | mgagne: but if you find a graph you like, feel free to propose it to the grafana dashboards | 17:34 |
| *** mrmartin has quit IRC | 17:34 | |
| *** mrmartin has joined #openstack-infra | 17:35 | |
| jeblair | mgagne: for instance, we could definitely add average runtime across all providers graph to grafana | 17:35 |
| *** samuelBartel has quit IRC | 17:35 | |
| *** jordanP has quit IRC | 17:35 | |
| mgagne | that's what I'm looking for but I think my grafana-fu isn't top, I'm getting 3y average | 17:35 |
| *** dslevin has joined #openstack-infra | 17:36 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add hue-light plugin support https://review.openstack.org/268814 | 17:36 |
| *** mrmartin has quit IRC | 17:37 | |
| *** kzaitsev_mb has quit IRC | 17:37 | |
| *** mrmartin has joined #openstack-infra | 17:38 | |
| *** _nadya_ has joined #openstack-infra | 17:38 | |
| openstackgerrit | Merged openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full https://review.openstack.org/268826 | 17:38 |
| sc68cal | I'm having difficulty pulling down patches from gerrit | 17:38 |
| *** daemontool has quit IRC | 17:38 | |
| sc68cal | http://paste.openstack.org/show/484192/ | 17:38 |
| *** daemontool has joined #openstack-infra | 17:39 | |
| fungi | sc68cal: did that happen just now? | 17:39 |
| *** peristeri has quit IRC | 17:39 | |
| sc68cal | fungi: yeah | 17:39 |
| sc68cal | ok this time it worked | 17:39 |
| fungi | or back before i restarted gerrit at 16:50 utc? | 17:39 |
| fungi | that's not a good sign | 17:40 |
| sc68cal | fungi: like 1 minute ago. Although looks like it's happy now | 17:40 |
| *** yamamoto has quit IRC | 17:40 | |
| *** nmagnezi has quit IRC | 17:40 | |
| pabelanger | jeblair: did we fix the regression in zuul yet? Curious if we can tag a new release | 17:40 |
| *** tonytan4ever has quit IRC | 17:40 | |
| fungi | zaro: ^ indication that this may not just be triggered by java gc busyness | 17:40 |
| pabelanger | sc68cal: I had a hiccup too with gerrit | 17:40 |
| fungi | javamelody claims gerrit is spending basically no time in gc (understandable since it was rebooted barely an hour ago) | 17:41 |
| *** mrmartin has quit IRC | 17:42 | |
| *** _nadya_ has quit IRC | 17:42 | |
| fungi | the javamelody system load graph indicates there was a significant spike in load average a few minutes ago, which probably corresponds to this | 17:42 |
| fungi | load shot up to 16 for one sample | 17:43 |
| AJaeger | could somebody approve the infra-manual change on retiring a project, please? https://review.openstack.org/#/c/210638/ | 17:44 |
| *** yaume has quit IRC | 17:44 | |
| anteaya | hey AJaeger | 17:46 |
| anteaya | fungi: what might cause it to spike? | 17:46 |
| *** sabeen3 has quit IRC | 17:46 | |
| *** tdurakov_ has joined #openstack-infra | 17:47 | |
| *** Sukhdev has quit IRC | 17:47 | |
| fungi | anteaya: good q. still looking around | 17:48 |
| fungi | anteaya: i'm betting on this, since the spikes line up http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=45&rra_id=all | 17:49 |
| mgagne | jeblair should I create a new dashboard? | 17:49 |
| anteaya | fungi: is someone cloning from gerrit? | 17:50 |
| fungi | anteaya: no clue what that was. could also have been a ton of api access i guess | 17:50 |
| *** yfried has joined #openstack-infra | 17:52 | |
| AJaeger | hi anteaya. | 17:54 |
| AJaeger | anteaya: just linked to 210638 when giving a -1 on https://review.openstack.org/269149 | 17:54 |
| *** hdd has joined #openstack-infra | 17:55 | |
| anteaya | AJaeger: nicely done | 17:56 |
| anteaya | I'm taking a peek at 269149 | 17:56 |
| AJaeger | thanks | 17:57 |
| anteaya | I agree with fungi that the implications to changes in the governance file do affect the electoral rolls | 17:57 |
| AJaeger | anteaya: I'd like to remove some jobs that fail completely, could you review, please? https://review.openstack.org/#/c/268067/ | 17:57 |
| AJaeger | anteaya: but wouldn't moving to the attic have the same problem? | 17:58 |
| anteaya | I wonder about creating a section somewhere in governance with names of projects that used to be official and are now attic, with a date on them so they can be removed | 17:58 |
| *** jaosorior has quit IRC | 17:59 | |
| anteaya | well as long as they were somewhere that they _were_ offical for some period of time for which their contributions could be attributed to atc status | 17:59 |
| AJaeger | can we merge the patch in some form and continue the discussion on this one point somehow? | 17:59 |
| *** Hal has joined #openstack-infra | 17:59 | |
| anteaya | let me look and think a bit | 17:59 |
| *** Hal is now known as Guest83643 | 18:00 | |
| *** tdurakov_ has quit IRC | 18:00 | |
| AJaeger | thanks | 18:00 |
| *** samuelBartel has joined #openstack-infra | 18:01 | |
| *** derekh has quit IRC | 18:03 | |
| *** zeih has quit IRC | 18:04 | |
| *** baoli has joined #openstack-infra | 18:04 | |
| *** tdurakov_ has joined #openstack-infra | 18:04 | |
| *** _nadya_ has joined #openstack-infra | 18:05 | |
| *** fawadkhaliq has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack-dev/hacking: Avoid git clone in integration test https://review.openstack.org/268275 | 18:07 |
| *** eil397 has joined #openstack-infra | 18:10 | |
| *** fawadkhaliq has joined #openstack-infra | 18:10 | |
| *** electrofelix has quit IRC | 18:10 | |
| *** fawadkhaliq has quit IRC | 18:11 | |
| AJaeger | with 268275 merged, https://review.openstack.org/#/c/268276/ is fine to merge now as well but needs some project-config review first if anybody wants, please | 18:11 |
| openstackgerrit | Merged openstack-infra/project-config: Remove translation from repos that are not setup https://review.openstack.org/268067 | 18:13 |
| *** yamahata has joined #openstack-infra | 18:15 | |
| *** tdurakov_ has quit IRC | 18:16 | |
| *** jistr has quit IRC | 18:17 | |
| *** Sukhdev has joined #openstack-infra | 18:17 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config: grafana: Add job average runtimes to Nodepool dashboard https://review.openstack.org/269232 | 18:18 |
| anteaya | AJaeger: my lunch is ready, still reviewing and thinking | 18:18 |
| mgagne | jeblair ^ | 18:18 |
| *** angdraug has quit IRC | 18:19 | |
| *** tdurakov_ has joined #openstack-infra | 18:20 | |
| *** yolanda has joined #openstack-infra | 18:21 | |
| *** trown|lunch is now known as trown | 18:22 | |
| *** dtantsur is now known as dtantsur|afk | 18:22 | |
| fungi | anteaya: i think it's probably better to just leave it up to the tc, with some guidance, as to whether or not removal of the repo is going to have an impact on elections. most of the removals so far have gone, like, a year with no commits (aside from the occasional mass update from one of our already established contributors for whom the loss of credit on that commit will make no difference to | 18:22 |
| fungi | qualifying as a voter) | 18:22 |
| fungi | i'll update my vote accordingly | 18:22 |
| *** davidlenwell has quit IRC | 18:22 | |
| *** david-lyle has quit IRC | 18:23 | |
| *** davidlenwell has joined #openstack-infra | 18:24 | |
| *** yaume has joined #openstack-infra | 18:27 | |
| *** [1]Thelo has joined #openstack-infra | 18:28 | |
| *** jpr2 has joined #openstack-infra | 18:29 | |
| *** dslevin has quit IRC | 18:30 | |
| *** Thelo has quit IRC | 18:30 | |
| *** [1]Thelo is now known as Thelo | 18:30 | |
| *** rossella_s has quit IRC | 18:30 | |
| *** rajinir has joined #openstack-infra | 18:31 | |
| *** rossella_s has joined #openstack-infra | 18:31 | |
| AJaeger | anteaya: sorry, I shouldn't have asked for review of that manual change with those -1s | 18:32 |
| *** jpr1 has quit IRC | 18:32 | |
| AJaeger | ah, thanks, fungi | 18:33 |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config: grafana: Add job average runtimes to Nodepool dashboard https://review.openstack.org/269232 | 18:33 |
| *** armax has quit IRC | 18:38 | |
| openstackgerrit | Morgan Fainberg proposed openstack/requirements: Add "positional" package to g-r https://review.openstack.org/267270 | 18:38 |
| *** armax has joined #openstack-infra | 18:39 | |
| anteaya | fungi: is there an item in front of the tc to this effect? | 18:39 |
| *** david-lyle has joined #openstack-infra | 18:39 | |
| fungi | anteaya: not directly, but every retirement which proposes removal from governance is at least acknowledged by the respective ptl (if the whole project is not completely abandoned) and put before the tc | 18:41 |
| fungi | so that still allows the tc to decide on a case by case basis | 18:41 |
| *** otter768 has joined #openstack-infra | 18:42 | |
| anteaya | okay | 18:43 |
| *** dslevin has joined #openstack-infra | 18:43 | |
| *** doug-fish has quit IRC | 18:43 | |
| *** hockeynut_afk is now known as hockeynut | 18:43 | |
| AJaeger | anteaya: we use merge-check for all active repos | 18:45 |
| AJaeger | anteaya: so the change to merge-check/noop is needed for removing all content from the repo | 18:46 |
| *** tdurakov_ has quit IRC | 18:46 | |
| AJaeger | anteaya: afterwards the entry can be removed completely | 18:46 |
| *** david-lyle has quit IRC | 18:46 | |
| *** otter768 has quit IRC | 18:46 | |
| anteaya | okay | 18:47 |
| *** dmsimard is now known as dmsimard|afk | 18:47 | |
| *** bardia has joined #openstack-infra | 18:47 | |
| anteaya | I guess I missed the part where it became obligatory | 18:48 |
| anteaya | rather than opt-in | 18:48 |
| AJaeger | anteaya: it's obligatory AFAIU | 18:48 |
| openstackgerrit | Merged openstack-infra/project-config: Remove glance python34 jobs on Liberty https://review.openstack.org/269178 | 18:48 |
| *** tdurakov_ has joined #openstack-infra | 18:48 | |
| openstackgerrit | Merged openstack-infra/project-config: Put ceilometer integration test on client lib https://review.openstack.org/267984 | 18:48 |
| anteaya | okay | 18:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-specs: Make translation set up consistent https://review.openstack.org/262545 | 18:49 |
| *** hashar has joined #openstack-infra | 18:51 | |
| *** annegentle has quit IRC | 18:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Create packetary project https://review.openstack.org/261540 | 18:51 |
| anteaya | the zuul job queue graph still looks very odd to me | 18:51 |
| AJaeger | yes - until the restart | 18:52 |
| openstackgerrit | Merged openstack-infra/project-config: Add jenkins job into steth project https://review.openstack.org/268847 | 18:54 |
| anteaya | then it just flatlined | 18:55 |
| zaro | fungi not really around today. But what was the purpose for gerrit restart? | 18:55 |
| anteaya | zaro: http://fungi.yuggoth.org/tmp/gc.png | 18:56 |
| anteaya | zaro: it was heavily into garbage collection again | 18:56 |
| *** mrmartin has joined #openstack-infra | 18:56 | |
| anteaya | zaro: also after the restart sc68cal had a 502: http://paste.openstack.org/show/484192/ | 18:56 |
| *** ashishb has joined #openstack-infra | 18:57 | |
| anteaya | zaro: so the 502's and 503's which had been attributed to the gc behaviour may not be | 18:57 |
| *** pvaneck has joined #openstack-infra | 18:57 | |
| mordred | nibalizer: I have verified - omfra cloud does support config-drive | 18:58 |
| mordred | nibalizer: you just have to request it when you boot an instance | 18:58 |
| *** yolanda has quit IRC | 18:59 | |
| nibalizer | oh neat | 18:59 |
| sc68cal | mordred: is that an API extension or something we offer | 18:59 |
| sc68cal | s/we/the openstack api/ | 18:59 |
| nibalizer | my nodepool is humming along, which is weird to me because I don't understand it | 19:00 |
| nibalizer | but investigation is proceeding :P | 19:00 |
| mordred | sc68cal: it's just a normal thing | 19:00 |
| sc68cal | bleh. | 19:00 |
| sc68cal | CAN WE SET THE MTU!?!?!?!? | 19:00 |
| mordred | nibalizer: we can also configure the cloud ot just always make a config drive by default | 19:01 |
| * sc68cal jumps from bridge | 19:01 | |
| mrmartin | hi | 19:01 |
| * mordred tosses sc68cal a life-vest that mostly doesn't have holes | 19:01 | |
| sc68cal | mordred: lol | 19:01 |
| mordred | hi mrmartin | 19:01 |
| *** burgerk has joined #openstack-infra | 19:02 | |
| *** Sukhdev has quit IRC | 19:02 | |
| *** Guest83643 has quit IRC | 19:03 | |
| *** jcoufal has quit IRC | 19:03 | |
| anteaya | sc68cal: I got through Sam-I-Am's dissertation, thanks for prompting the conversation | 19:03 |
| *** Sukhdev has joined #openstack-infra | 19:03 | |
| anteaya | I wonder how many of our ssh bugs are related | 19:03 |
| AJaeger | anteaya: can you change Sam-I-Am's nic to Dr-I-Am, please ;) | 19:03 |
| mrmartin | may I get a little care to this patch: https://review.openstack.org/#/c/265875/ This will help to bring back askbot-staging alive, and I found the bug and contributed a fix to upstream askbot, so the openstackid.org login would go to production. | 19:03 |
| *** sabeen has joined #openstack-infra | 19:05 | |
| *** tdurakov_ has quit IRC | 19:05 | |
| *** rockyg has joined #openstack-infra | 19:05 | |
| sc68cal | anteaya: AJaeger: +1 Sam-I-Am is awesome | 19:06 |
| anteaya | AJaeger: :) | 19:06 |
| anteaya | sc68cal: agreed, Sam-I-Am is awesome | 19:06 |
| *** sabeen3 has joined #openstack-infra | 19:06 | |
| fungi | zaro: increased time spent in gc, heavy system load, elevated cpu utilization, frequent http 502 errors from webui and rest api | 19:07 |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: one function for updating users passwords https://review.openstack.org/268361 | 19:07 |
| *** kzaitsev_mb has joined #openstack-infra | 19:08 | |
| *** _nadya_ has quit IRC | 19:09 | |
| *** sabeen has quit IRC | 19:09 | |
| *** baoli has quit IRC | 19:10 | |
| *** david-lyle has joined #openstack-infra | 19:10 | |
| *** tdurakov_ has joined #openstack-infra | 19:11 | |
| *** gfreeman has joined #openstack-infra | 19:11 | |
| *** flepied1 has joined #openstack-infra | 19:11 | |
| *** tonytan4ever has joined #openstack-infra | 19:12 | |
| *** ociuhandu has quit IRC | 19:12 | |
| *** flepied has quit IRC | 19:13 | |
| *** sfinucan has quit IRC | 19:14 | |
| *** achanda has quit IRC | 19:14 | |
| *** dslevin has quit IRC | 19:18 | |
| *** peristeri has joined #openstack-infra | 19:19 | |
| *** abregman has joined #openstack-infra | 19:19 | |
| *** lennyb has quit IRC | 19:20 | |
| *** lennyb has joined #openstack-infra | 19:20 | |
| *** abregman is now known as abregman|nb | 19:20 | |
| krotscheck | Do we have statistics about job failure rate by job name? | 19:21 |
| *** akshai_ has quit IRC | 19:22 | |
| mgagne | krotscheck I think we have http://status.openstack.org/openstack-health/#/ | 19:22 |
| *** baoli has joined #openstack-infra | 19:23 | |
| mgagne | krotscheck I think you can dig into projects to get more info and list of jobs | 19:23 |
| krotscheck | mgagne: I see that, awesome. | 19:23 |
| *** kzaitsev_mb has quit IRC | 19:26 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: one function for updating users passwords https://review.openstack.org/268361 | 19:26 |
| openstackgerrit | Merged openstack-infra/system-config: Pin ask staging branch to 0.7.x and update settings https://review.openstack.org/265875 | 19:26 |
| anteaya | changes to nodepool elements scripts, do they go into effect with the next image build? motivation for my question: https://review.openstack.org/#/c/268841/2 | 19:27 |
| *** david-lyle_ has joined #openstack-infra | 19:27 | |
| *** dslev has joined #openstack-infra | 19:28 | |
| *** baoli has quit IRC | 19:28 | |
| *** david-lyle has quit IRC | 19:31 | |
| *** baoli has joined #openstack-infra | 19:32 | |
| *** salv-orlando has joined #openstack-infra | 19:32 | |
| *** akshai has joined #openstack-infra | 19:35 | |
| *** fawadkhaliq has joined #openstack-infra | 19:35 | |
| anteaya | gerrit just failed to take my review, it said working... and then my review disappeared | 19:36 |
| anteaya | second time lucky | 19:36 |
| *** david-lyle_ has quit IRC | 19:37 | |
| *** david-lyle has joined #openstack-infra | 19:39 | |
| *** baoli has quit IRC | 19:40 | |
| *** dmsimard|afk is now known as dmsimard | 19:40 | |
| *** achanda has joined #openstack-infra | 19:40 | |
| *** baoli has joined #openstack-infra | 19:41 | |
| *** mrmartin has quit IRC | 19:41 | |
| *** baoli has quit IRC | 19:42 | |
| *** yaume has quit IRC | 19:43 | |
| *** zeih has joined #openstack-infra | 19:44 | |
| *** david-lyle has quit IRC | 19:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystone starting with mitaka https://review.openstack.org/261994 | 19:49 |
| *** _nadya_ has joined #openstack-infra | 19:49 | |
| *** ifarkas has quit IRC | 19:50 | |
| adam_g | mordred, if you're not holidayin'.. had to rebase this to fix a conflict over the weekend https://review.openstack.org/#/c/268285/ | 19:50 |
| krotscheck | Anybody want a wheel mirror? I know you do! https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open+project:openstack-infra/system-config | 19:50 |
| *** salv-orlando has quit IRC | 19:51 | |
| *** achanda has quit IRC | 19:53 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:54 | |
| *** fawadkhaliq has quit IRC | 19:56 | |
| *** moravec has quit IRC | 19:56 | |
| *** e0ne has joined #openstack-infra | 19:56 | |
| *** bcourt has quit IRC | 20:03 | |
| clarkb | there is no elasticsearch process running on es02 | 20:03 |
| clarkb | dmesg reports possible syn flooding on es ports | 20:04 |
| clarkb | I think the cluster must've gone crazy for some reason and tried to rebuild itself then failed and turned off | 20:04 |
| clarkb | but the logs don't really say much | 20:04 |
| *** baoli has joined #openstack-infra | 20:05 | |
| jeblair | clarkb: looks like a couple days ago: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=1651&rra_id=2&view_type=&graph_start=1452850841&graph_end=1452926668&graph_height=120&graph_width=500&title_font_size=10 | 20:06 |
| *** _nadya_ has quit IRC | 20:06 | |
| clarkb | the 14th? | 20:06 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 20:06 |
| clarkb | I was going to look and see if rax reported anything then since that was when it last logged anything | 20:07 |
| ianw | clarkb: (when you're finished with elasticsearch or whatever) did you see the --prune stuff for early devstack clone in https://review.openstack.org/#/c/268841/ ? i also stripped the whole thing down in https://review.openstack.org/268888 | 20:07 |
| clarkb | ianw: I hadn't seen your changes yet. I did make a change to dib that should fix it though | 20:08 |
| anteaya | clarkb: :( | 20:08 |
| anteaya | ianw: I was looking at those | 20:08 |
| clarkb | ianw: so the prune is not sufficient | 20:08 |
| anteaya | ianw: I haven't done much reviewing in nodepool/elements, do changes in those scripts get picked up in the next image build? | 20:08 |
| clarkb | ianw: all the prune does is update your remote branches it doesn't delete stable/foo if origin/stable/foo is pruned | 20:09 |
| AJaeger | anteaya: yes | 20:09 |
| anteaya | AJaeger: thank you | 20:09 |
| ianw | anteaya: yep | 20:09 |
| clarkb | ianw: so we still need https://review.openstack.org/#/c/268241/ in dib generally, thoug hI agree that in this particular instance our forked element is the larger problem | 20:10 |
| *** baoli has quit IRC | 20:10 | |
| *** yaume has joined #openstack-infra | 20:10 | |
| anteaya | ianw: thank you | 20:10 |
| *** eil397 has quit IRC | 20:13 | |
| *** tdurakov_ has quit IRC | 20:13 | |
| *** angdraug has joined #openstack-infra | 20:14 | |
| clarkb | looks like we got to the bottom of the zuul job graph? what was going on there? | 20:14 |
| anteaya | clarkb: in the commit message, I can't follow the first sentence in the second paragraph: https://review.openstack.org/#/c/268241/ | 20:14 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/system-config: Fix regex in NoRoaming workaround https://review.openstack.org/269258 | 20:14 |
| anteaya | the zuul job graph looks flatlined to me | 20:14 |
| clarkb | anteaya: I can try rewriting it | 20:15 |
| *** tdurakov_ has joined #openstack-infra | 20:15 | |
| anteaya | which is not what I expect the zuul job graph to look like | 20:15 |
| anteaya | clarkb: thanks | 20:15 |
| clarkb | except new ui doesn't seem to let me edit commit message I guess I should use igt | 20:15 |
| anteaya | if someone knows what is happening with the zuul job graph, I haven't picked up on their understanding if they have tried to communicate it | 20:15 |
| anteaya | clarkb: it is supposed to | 20:16 |
| *** ddieterly has quit IRC | 20:16 | |
| SamYaple | clarkb: you can edit it | 20:16 |
| anteaya | that is a feature of new ui | 20:16 |
| clarkb | but how | 20:16 |
| *** ldnunes has quit IRC | 20:16 | |
| anteaya | https://review.openstack.org/#/c/268241/3//COMMIT_MSG | 20:16 |
| clarkb | there was a button next to the commit mesage to edit it in the past | 20:16 |
| anteaya | patchset 1 2 3 edit button | 20:16 |
| anteaya | I know | 20:16 |
| anteaya | this is harder to find | 20:16 |
| clarkb | oh I have to open the thing in the diff? | 20:16 |
| anteaya | yes | 20:16 |
| anteaya | silly silly silly | 20:16 |
| AJaeger | clarkb: now you can change *each* file | 20:17 |
| *** dprince has quit IRC | 20:17 | |
| ianw | clarkb: so i don't think local branches are the issue. have a look at http://nodepool.openstack.org/image.log and search for "14:28:17,336" ... cache-devstack-repos is picking up remotes/origin/stable/... | 20:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystone starting with mitaka https://review.openstack.org/269261 | 20:18 |
| *** bcourt has joined #openstack-infra | 20:18 | |
| clarkb | this is confusing | 20:18 |
| clarkb | ianw: they are an issue because there is some striaght file copying iirc rather than git cloning happening | 20:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka https://review.openstack.org/269261 | 20:19 |
| mmedvede | does anyone know why there are no new logs on http://nodepool.openstack.org? Most recent ones are from 3 days ago. I was checking if puppet apply was working before submitting https://review.openstack.org/269258 | 20:19 |
| clarkb | ianw: if we used git clone everywhere we would have fewer problems likely | 20:19 |
| *** tonytan4ever has quit IRC | 20:19 | |
| clarkb | so I made edits and saved them but no new patchset | 20:19 |
| clarkb | oh you have to go out a window and click "done editing" | 20:19 |
| anteaya | mmedvede: well possibly because image build have failed for the last few days | 20:19 |
| anteaya | clarkb: yes and save and really save | 20:19 |
| clarkb | uh still no new patchset | 20:20 |
| clarkb | I give up will make changes locally | 20:20 |
| anteaya | clarkb: must have missed a save somewhere | 20:20 |
| ianw | mmedvede: hmm, i bet it's related to the builders change | 20:20 |
| clarkb | (what good are feature slike this if they are completely unuseable | 20:20 |
| SamYaple | clarkb: thats the spirit! | 20:20 |
| *** ddieterly has joined #openstack-infra | 20:20 | |
| anteaya | sorry to distract from the real issue here, commit message can wait | 20:20 |
| SamYaple | clarkb: its a pain i agree. all i can say is it can be done | 20:20 |
| clarkb | ianw: mmedvede we did have working builds last week they just happened to fail | 20:20 |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: allow for updating passwords in keystone v2 https://review.openstack.org/268361 | 20:21 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Actually prune stale branches in src-repos https://review.openstack.org/268241 | 20:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystone starting with mitaka https://review.openstack.org/261994 | 20:21 |
| ianw | clarkb: no, i feel like the logging config in system-config is not matching on the logger for dib builds now, and the logs aren't getting put into nodepool.openstack.org's directory | 20:21 |
| clarkb | anteaya: ^ hopeflly the extra punctuation makes it readable | 20:21 |
| AJaeger | clarkb: there's a publish edits | 20:21 |
| clarkb | ianw: ah right I think pabelanger did mention that | 20:21 |
| anteaya | clarkb: thank you, yes I can read that now | 20:21 |
| clarkb | so we just need to update the logging config | 20:21 |
| AJaeger | clarkb: it's different and non-intuitive -but if you get the hang for it, it works nice | 20:22 |
| ianw | pabelanger: ^ did you already do a fix that i missed? | 20:22 |
| clarkb | looks like dfw did have planned maintenance on networking | 20:23 |
| *** annegentle has joined #openstack-infra | 20:23 | |
| clarkb | its not clear if they actually performed such things | 20:23 |
| anteaya | clarkb: that might have affected es02? | 20:23 |
| clarkb | anteaya: yes (and 06 was off too) | 20:23 |
| clarkb | anteaya: basically what happens when nodes can't talk to each other for a prolonged period of time is they give up | 20:24 |
| anteaya | mriedem: logstash is back | 20:24 |
| clarkb | so rather than run in a known broken state they just turn off | 20:24 |
| mriedem | yup, just noticed | 20:24 |
| anteaya | clarkb: I can understand that with elastic search | 20:24 |
| anteaya | better than taking out a vm I guess | 20:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka https://review.openstack.org/269261 | 20:24 |
| anteaya | clarkb: thanks for making logstash come back | 20:25 |
| *** tonytan4ever has joined #openstack-infra | 20:25 | |
| ianw | clarkb: hmm, no "nodepool.image.build.*" ( https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/builder.py#n353 ) should get caught and sent to public log files. can you please put in your queue to check the private log file and see if build logs are bubbling into there? | 20:26 |
| clarkb | can do | 20:27 |
| ianw | ahhhh ... unless the builder doesn't read the log config file... | 20:27 |
| *** dtardivel has quit IRC | 20:27 | |
| *** doug-fish has joined #openstack-infra | 20:28 | |
| ianw | which it doesn't | 20:28 |
| clarkb | its still in procedd though | 20:28 |
| anteaya | so 268841 and 268241, are both patches needed or is it one or the other? | 20:29 |
| pabelanger | clarkb: ianw : Yup: https://review.openstack.org/#/c/267865/ | 20:30 |
| pabelanger | like we do for zuul | 20:30 |
| *** gildub has joined #openstack-infra | 20:31 | |
| ianw | pabelanger: great, i'll review soon | 20:32 |
| pabelanger | ianw: great | 20:32 |
| clarkb | ianw: grepping for image in the debug log doesn't show any of the builds logs | 20:32 |
| ianw | clarkb: but yeah, the builder process isn't setting up any logging? but pabelanger's change should fix taht | 20:33 |
| clarkb | ianw: there is no separate builder process | 20:34 |
| clarkb | (not yet) | 20:34 |
| ianw | i thought it got merged, sorry, i'm a bit out of the loop on this bit | 20:34 |
| clarkb | it did, but it can run in process or out of process | 20:35 |
| clarkb | we are currently running in process because that was the minimal amount of change to get it working and work through the issues | 20:35 |
| clarkb | avoided needing like 3 changes to merge all at once too | 20:35 |
| ianw | oh, ok, there's a sneaky self.builder = nodepool.builder.NodePoolBuilder(self.args.config) in nodepoold | 20:36 |
| *** vgridnev has joined #openstack-infra | 20:36 | |
| ianw | then why have logs disappeared ... | 20:36 |
| *** flwang has joined #openstack-infra | 20:37 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 20:37 | |
| *** hashar has quit IRC | 20:37 | |
| clarkb | not sure yet but I definitely don't see them in the main debug log so I don't think its just a matter of paths changing | 20:37 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:37 | |
| ianw | hmm, no. setup_logging has still run, so that seems all ok | 20:37 |
| clarkb | we should double check the devstack job to see that they are in there | 20:38 |
| clarkb | as that is a simpler setup | 20:38 |
| *** markvoelker has joined #openstack-infra | 20:38 | |
| *** lucasagomes is now known as lucas-dinner | 20:39 | |
| ianw | i wonder if it's something to do with "import nodepool.builder" *after* setup_logging | 20:39 |
| clarkb | maybe? though nodepool.nodepool is imported late too | 20:40 |
| *** weshay has quit IRC | 20:40 | |
| ianw | but *before* the setup_logging | 20:40 |
| clarkb | ah | 20:40 |
| ianw | i think we can get rid of these late imports with https://review.openstack.org/#/c/267885/ | 20:41 |
| clarkb | easy way to check is push a change that flips the order | 20:41 |
| clarkb | and yes there is a change to fix the statsd stuff | 20:41 |
| ianw | although i'm not 100% sure about the comments on paramiko | 20:41 |
| clarkb | so maybe just review that | 20:41 |
| clarkb | with paramkiko we may have to whitelist its fds | 20:42 |
| *** otter768 has joined #openstack-infra | 20:42 | |
| clarkb | ianw: the statsd change you wrote lgtm but we need to undo the late import for builders in it or in a follow up change (since my workaround change merged first) | 20:44 |
| clarkb | ianw: should I go ahead and ervert my change and we merge both together? | 20:44 |
| ianw | clarkb: yeah, well i wasn't sure about a full cleanup because of that paramiko comment. it seems like we could just do the imports at the top as you say, if you think it's safe? | 20:45 |
| clarkb | for paramiko I think we definitely need to do something more like whitelist the fds | 20:45 |
| clarkb | for the builders I think we are ok as is ? let me check that it odesn't import paramiko too | 20:45 |
| clarkb | ya no paramiko there | 20:46 |
| adam_g | anyone around to +W a requirements patch that needed a rebase over the weekend? thanks! https://review.openstack.org/#/c/268285/ | 20:46 |
| ianw | ahh, ok, then yeah we should be able to do builders. i can put a quick stack together for that | 20:46 |
| clarkb | ianw: for builders it was just statsd which yo uaddressed | 20:47 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka https://review.openstack.org/269261 | 20:47 |
| *** otter768 has quit IRC | 20:47 | |
| annegentle | anyone have hints or tips for changing the email address associated with Launchpad's login/openid? Just changing it in email settings doesn't let me log in with that email address on login.launchpad.net. | 20:47 |
| *** ashishb has quit IRC | 20:47 | |
| *** zeih has quit IRC | 20:47 | |
| clarkb | annegentle: you may hav to do it through the ubuntu one interface | 20:50 |
| *** vivekd has quit IRC | 20:50 | |
| annegentle | clarkb: ah ok | 20:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Don't use global statsd https://review.openstack.org/267885 | 20:51 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Revert "Import nodepool.builder late for statsd" https://review.openstack.org/269266 | 20:51 |
| ianw | clarkb / mgagne / pabelanger: ^ i suspect that might get logs routing again, of course i haven't tested it to ensure that is true :) | 20:51 |
| clarkb | ianw: the devstack run should tell us | 20:52 |
| clarkb | mayb | 20:52 |
| ianw | pabelanger: can we shoehorn that into your nodepool instance? | 20:52 |
| *** ZZelle_ has joined #openstack-infra | 20:52 | |
| clarkb | nibalizer: hvae you have a chance to update your puppet for the nodepool builders? | 20:53 |
| *** ihrachys has joined #openstack-infra | 20:53 | |
| nibalizer | clarkb: this got updated | 20:54 |
| nibalizer | https://review.openstack.org/#/c/240093/ | 20:54 |
| nibalizer | i need to fix tis https://review.openstack.org/#/c/240094/ | 20:54 |
| *** weshay has joined #openstack-infra | 20:55 | |
| *** annegentle has quit IRC | 20:55 | |
| *** amotoki has joined #openstack-infra | 20:55 | |
| *** annegentle has joined #openstack-infra | 20:55 | |
| mordred | nibalizer: HEYA! | 20:55 |
| mordred | nibalizer: you wanna review some ansibles? | 20:55 |
| nibalizer | heyyyyy | 20:56 |
| nibalizer | mordred: sure | 20:56 |
| nibalizer | are we 2.0ing | 20:56 |
| *** hichihara has joined #openstack-infra | 20:56 | |
| mordred | nibalizer: https://review.openstack.org/#/q/topic:puppet-apply+status:open is sitting over in the corner waiting for you to buy it a drink | 20:56 |
| nibalizer | clarkb: its annoying but I think we probably have to pipe it all through openstackci | 20:57 |
| *** baoli has joined #openstack-infra | 20:57 | |
| *** vgridnev has quit IRC | 20:58 | |
| clarkb | nibalizer: couldn't we just turn it on | 20:58 |
| clarkb | ? | 20:58 |
| clarkb | I am wondering if we need to make it toggleable now that the nodepool code has merged | 20:58 |
| clarkb | maybe tag old nodepool and old puppet if we want to designate the cutoff | 20:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use linters job for keystoneclient starting with mitaka https://review.openstack.org/269270 | 20:59 |
| nibalizer | clarkb: in 240093 we created a nodepool::builder class, we need to instantiate that somewhere | 20:59 |
| nibalizer | and I think the polite thing is to pipe that through an openstackci class | 21:00 |
| clarkb | right so openstackci should do it but I see on the sytem-config side we are passing a flag through which makes it a little more complicated | 21:00 |
| clarkb | just wondering if we need the flag | 21:00 |
| *** baoli has quit IRC | 21:00 | |
| *** eil397 has joined #openstack-infra | 21:02 | |
| stevemar | AJaeger: you have all this on the same topic branch so i do have to go searching through reviews? :) | 21:02 |
| nibalizer | clarkb: thats a really old patch | 21:03 |
| *** ihrachys_ has joined #openstack-infra | 21:04 | |
| *** eil397 has quit IRC | 21:04 | |
| *** eil397 has joined #openstack-infra | 21:05 | |
| clarkb | yes well I thought we could get this in in tokyo :) | 21:05 |
| stevemar | AJaeger: ah there they all are: https://review.openstack.org/#/q/topic:merge-bandit | 21:06 |
| *** yaume has quit IRC | 21:06 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Create a new openstack project https://review.openstack.org/269272 | 21:07 |
| *** ihrachy__ has joined #openstack-infra | 21:07 | |
| *** ihrachy__ is now known as ihrachysh | 21:07 | |
| Clint | jeblair: i seem to have an endless-loop-in-syncing problem in gertty.. thoughts? | 21:07 |
| *** apoorvad has quit IRC | 21:08 | |
| *** ihrachys has quit IRC | 21:08 | |
| *** daemontool has quit IRC | 21:08 | |
| *** ihrachys_ has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack-infra/project-config: grafana: Add job average runtimes to Nodepool dashboard https://review.openstack.org/269232 | 21:12 |
| *** achanda has joined #openstack-infra | 21:12 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers https://review.openstack.org/240093 | 21:14 |
| anteaya | Clint: he might suggest you running it in debug mode and paste the log | 21:14 |
| anteaya | he often makes that suggestion | 21:14 |
| Clint | anteaya: sensible | 21:15 |
| anteaya | he's like that | 21:15 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers https://review.openstack.org/240093 | 21:16 |
| *** achanda has quit IRC | 21:17 | |
| *** tonytan4ever has quit IRC | 21:20 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackci: Consume nodepool::builder class https://review.openstack.org/269280 | 21:22 |
| nibalizer | clarkb: 269280 is one way of doing it | 21:23 |
| clarkb | I don't see any recent images in nodepool, so it may be failing to run the builds at all explaining the lack of logs | 21:23 |
| clarkb | I did confirm that manually triggered builds were running and that dib ran it just couldn't succeed | 21:23 |
| clarkb | so need to dig into thtat more | 21:23 |
| clarkb | greghaynes: ^ | 21:23 |
| nibalizer | clarkb: are you wanting to colocate a builder today or start spinning up a little pool of builders? | 21:23 |
| clarkb | colocate is what we will probably start with | 21:24 |
| nibalizer | I'd kind of prefer to go full worker pool rather than the current stuff in 269280 | 21:24 |
| clarkb | the problem with jumping to a full worker pool is nothing is working right now for a variety of reasons | 21:24 |
| clarkb | so it will be hard to separate new fails from old fails | 21:24 |
| nibalizer | haha ok | 21:24 |
| greghaynes | clarkb: lol | 21:25 |
| nibalizer | also openstackci is built on the assumption of total colocation so thats easy | 21:25 |
| clarkb | we can't build dib images | 21:25 |
| greghaynes | clarkb: fun | 21:25 |
| *** vgridnev has joined #openstack-infra | 21:25 | |
| clarkb | snapshots don't appear to be updating either | 21:25 |
| clarkb | so I am going to try a manual trigger | 21:25 |
| greghaynes | Hrm | 21:25 |
| greghaynes | So I suspect that isn't me then unless there is some tracebacks | 21:25 |
| *** kgiusti has left #openstack-infra | 21:26 | |
| clarkb | the dib thing isn't | 21:26 |
| clarkb | but the lack of logs and lack of new snapshot images is suspicious | 21:26 |
| greghaynes | clarkb: ya | 21:26 |
| clarkb | which reminds me to vote on the statsd changes | 21:26 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper https://review.openstack.org/269284 | 21:28 |
| clarkb | manually triggered image builds are doing things | 21:29 |
| *** keedya_ has quit IRC | 21:33 | |
| *** abregman|nb has quit IRC | 21:33 | |
| *** Keedya_ has joined #openstack-infra | 21:34 | |
| mordred | clarkb: I look forward to learning things | 21:35 |
| mordred | nibalizer: I'd love to land https://review.openstack.org/#/c/265647/ so that I can verify that it's copying the right files | 21:37 |
| mordred | clarkb: also, I started work on getting the nodepool-shade patch up to speed | 21:37 |
| mordred | clarkb: the patch that's up in gerrit is not ready - but we're looking good | 21:37 |
| clarkb | mordred: is it the same change as before just updated? | 21:38 |
| mordred | clarkb: mostly | 21:38 |
| mordred | clarkb: there are a few differences, and I have a few more locally | 21:38 |
| *** tonytan4ever has joined #openstack-infra | 21:38 | |
| *** Keedya_ has quit IRC | 21:38 | |
| mordred | clarkb: because we added several things to shade since last we updated that patch based on things we needed from that work | 21:38 |
| clarkb | gotcha | 21:38 |
| mordred | so going back through and actually taking advantage of all of those things is what I'm doing currently - when I'm not trying to convince people to review puppet-apply patches :) | 21:39 |
| greghaynes | clarkb: I put up a nodepool image build failure fix too, btw | 21:39 |
| clarkb | greghaynes: for the uncatching of fails? | 21:39 |
| greghaynes | clarkb: I kind of want to write a test case | 21:39 |
| * clarkb goes to review that one | 21:39 | |
| greghaynes | Yes | 21:39 |
| *** yaume has joined #openstack-infra | 21:40 | |
| clarkb | then if we can get the statsd fix, logging fix, and fail fixes in before next restart we should be in good shape to make sure things are working again | 21:40 |
| clarkb | greghaynes: looks like the dsvm jobs timed out so I have rechceked it | 21:40 |
| *** apoorvad has joined #openstack-infra | 21:42 | |
| clarkb | greghaynes: for testing that you can setup an image build fail via the env vars for the image. We do this on at least one other image so probably that test isn't strong enough | 21:42 |
| *** apoorvad has quit IRC | 21:42 | |
| clarkb | but maybe we need a new test too | 21:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config: zuul_swift_upload add support for --verbose https://review.openstack.org/269288 | 21:42 |
| *** apoorvad has joined #openstack-infra | 21:43 | |
| greghaynes | clarkb: yep, there is an image build fail test case but obviously it doesn't test enough | 21:43 |
| greghaynes | clarkb: I just haven't had time to look at it yet | 21:43 |
| clarkb | greghaynes: test_dib_and_snap_fail is the existing test | 21:43 |
| clarkb | ok the assertion is just that no nodes build | 21:44 |
| anteaya | crinkle: thanks for the etherpad | 21:44 |
| clarkb | so can be tightened up | 21:44 |
| greghaynes | clarkb: ya, easy enough | 21:44 |
| jhesketh | Morning | 21:45 |
| ianw | clarkb: so from what i can see, the issue with paramiko came down to creting a global Crypto.Random() object which was created at import time. if you daemonize after that import it's fd for /dev/urandom got killed | 21:46 |
| ianw | clarkb: however, it seems like they don't do that any more -> https://github.com/paramiko/paramiko/commit/6f211115f49edcea7d23b764d7cf3a84ff12f5f0 | 21:46 |
| anteaya | morning jhesketh | 21:46 |
| ianw | clarkb: ergo, i think with a bit of testing/version checking we can remove that bit too | 21:47 |
| clarkb | ianw: woot | 21:47 |
| crinkle | anteaya: no problem | 21:48 |
| *** yaume has quit IRC | 21:49 | |
| *** weshay has quit IRC | 21:49 | |
| clarkb | ianw: just need to figure out what the first release that fixed that is and don't allow anything older in requirements | 21:50 |
| clarkb | ianw: then the change should be self testing in the dsvm jobs since they ssh into real VMs and do all that | 21:50 |
| *** shashank_hegde has joined #openstack-infra | 21:50 | |
| clarkb | ianw: so its easy to have the computer check that it works for us if you want ot go ahead and push the change to bump the min version | 21:50 |
| *** jtomasek has quit IRC | 21:50 | |
| *** shardy has quit IRC | 21:51 | |
| *** ericksonsantos has quit IRC | 21:51 | |
| openstackgerrit | Merged openstack-infra/system-config: Add script to make ansible groups from patterns https://review.openstack.org/265647 | 21:52 |
| ianw | yeah, 1.11.6 is the first after that commit ... good place to start | 21:52 |
| *** jflynn has joined #openstack-infra | 21:53 | |
| *** burgerk has quit IRC | 21:53 | |
| *** vgridnev has quit IRC | 21:53 | |
| nibalizer | mordred: im curious where we picked up 'hieraenvironment' | 21:53 |
| nibalizer | there are only puppet environments | 21:53 |
| nibalizer | hiera doesn't have them | 21:53 |
| mordred | nibalizer: it's a parameter | 21:55 |
| mordred | nibalizer: possibly poorly named | 21:55 |
| mordred | nibalizer: should we call it puppet environment instead? | 21:55 |
| nibalizer | yes | 21:55 |
| nibalizer | not the most important | 21:55 |
| nibalizer | but wiggs me every time I see it | 21:55 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Add puppet module sync to the role https://review.openstack.org/251580 | 21:56 |
| clarkb | manual snapshot build looks already so we may just need fixed logging in order to diagnose the daemonized stuff | 21:56 |
| clarkb | I have +2'd the two changes to do that | 21:56 |
| mordred | nibalizer: k. then we should update it fo sho | 21:56 |
| clarkb | but should confirm in the dsvm logs that it looks correct | 21:57 |
| *** tdurakov_ has quit IRC | 21:57 | |
| *** bcourt has quit IRC | 21:58 | |
| mordred | nibalizer: I'll send in a follow up patch | 22:00 |
| *** bknudson_ has quit IRC | 22:02 | |
| *** kzaitsev1mb has joined #openstack-infra | 22:02 | |
| *** jordanP has joined #openstack-infra | 22:02 | |
| *** edmondsw has quit IRC | 22:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Rename hieraenvironment to environment https://review.openstack.org/269299 | 22:02 |
| mordred | nibalizer: ^^ thereyago | 22:02 |
| *** kzaitsev_mb has quit IRC | 22:02 | |
| *** weshay has joined #openstack-infra | 22:03 | |
| clarkb | mordred: https://review.openstack.org/#/c/265647/ should come with a documentation update since that is something roots have to know to do | 22:04 |
| *** jpr2 has quit IRC | 22:06 | |
| nibalizer | mordred: for that, is environment really not a resereved word in ansible | 22:06 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: use ceilometerclient from git https://review.openstack.org/269302 | 22:07 |
| *** moravec has joined #openstack-infra | 22:07 | |
| mordred | nibalizer: oh! yes it is. | 22:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Remove workaround for early paramiko import https://review.openstack.org/269303 | 22:08 |
| mordred | nibalizer: gross. I guess we have to call it "puppet_environmnet" then? | 22:08 |
| nibalizer | ya | 22:08 |
| nibalizer | what is the default ansible environment | 22:09 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Delete old queries for fixed bugs https://review.openstack.org/269305 | 22:09 |
| *** jamielennox|away is now known as jamielennox | 22:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Rename hieraenvironment to puppetenvironment https://review.openstack.org/269299 | 22:09 |
| *** gildub has quit IRC | 22:10 | |
| nibalizer | mordred: hrm so in https://review.openstack.org/#/c/243399/12/defaults/main.yml i've been doing puppet_thing for config directives | 22:10 |
| *** bcourt has joined #openstack-infra | 22:11 | |
| *** ociuhandu has joined #openstack-infra | 22:11 | |
| mordred | nibalizer: I hate you | 22:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Rename hieraenvironment to puppet_environment https://review.openstack.org/269299 | 22:12 |
| *** chlong has joined #openstack-infra | 22:12 | |
| mordred | nibalizer: maybe this time? :) | 22:12 |
| fungi | wow. puppet_thing is an element in the puppet_stuff list, right? | 22:12 |
| * fungi is hoping | 22:12 | |
| nibalizer | fungi: naturally | 22:12 |
| clarkb | oh I know the images that failed are still in "building" state in the db | 22:13 |
| clarkb | so nodepool says "I have nothing to do" | 22:13 |
| mordred | clarkb: woot | 22:13 |
| clarkb | so ya we should get greghaynes' fix in | 22:13 |
| fungi | ahh, because of the failures not updating the db issue? | 22:13 |
| *** regXboi has quit IRC | 22:13 | |
| clarkb | fungi: yup | 22:13 |
| fungi | clarkb: greghaynes: what's the patch number on that? | 22:13 |
| clarkb | fungi: https://review.openstack.org/#/c/268867/1 | 22:13 |
| clarkb | I rechecked it to see if the dsvm fails are consistent | 22:13 |
| fungi | appreciated | 22:13 |
| clarkb | looks like they timed out | 22:13 |
| clarkb | whcih may be just due to bluebox | 22:14 |
| *** bpokorny has joined #openstack-infra | 22:14 | |
| fungi | the source one ran in ovh | 22:14 |
| fungi | but both hit the timeout | 22:15 |
| clarkb | it looks like 1 of 2 of the VMs we boot managed to start | 22:15 |
| *** peristeri has quit IRC | 22:16 | |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Update to use blocks from ansible 2.0 https://review.openstack.org/266361 | 22:16 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Clean up codestyle in conditionals https://review.openstack.org/266369 | 22:16 |
| *** dims has joined #openstack-infra | 22:17 | |
| clarkb | (I am just happy I got to the bottom of why no builds were happening) | 22:17 |
| clarkb | I am tempted to delete them fro mthe db to confirm we get new builds tomorrow, but those will fail so \o/ | 22:17 |
| *** treaki_ has quit IRC | 22:17 | |
| *** eil397 has quit IRC | 22:17 | |
| clarkb | I guess I should get ianw's prune fix for project-config in and maybe thats sufficient | 22:17 |
| mordred | clarkb, fungi: is there any way to get emacs to not write backup files in a given directory | 22:17 |
| mordred | ? | 22:17 |
| *** dims_ has quit IRC | 22:18 | |
| clarkb | mordred: emacs? | 22:18 |
| mordred | yeah - that foo~ files | 22:18 |
| clarkb | (I honestly don't know, too much of a vim user) | 22:18 |
| *** dkranz has quit IRC | 22:18 | |
| mordred | currently there is a file - /etc/ansible/hosts/emergency~ | 22:18 |
| fungi | mordred: you want to change that behavior on a per-directory way? | 22:19 |
| *** julim has quit IRC | 22:19 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Needed-By: I9fa0ae1831baf6c17ea58d517b77cbc166fa408f https://review.openstack.org/269308 | 22:19 |
| craige | o/ | 22:19 |
| mordred | fungi: maybe? | 22:20 |
| fungi | mordred: you can tell emacs to put them all somewhere else | 22:20 |
| ianw | clarkb: i'm still trying to convince myself too, i don't feel like i fully understand why we're getting those branches there | 22:20 |
| mordred | ZOMG I WISH I COULD FIGURE OUT WHY PUPPETMASTER HATES ME | 22:20 |
| clarkb | ianw: ++ it doesn't hurt to have that change in there so I have approved it. Reviewing the script simplification too | 22:20 |
| mordred | fungi: for some reason, my ssh connections time out fairly constantly - _only_ to puppetmaster.o.o | 22:20 |
| mordred | which confuses me | 22:21 |
| *** Keedya_ has joined #openstack-infra | 22:21 | |
| fungi | mordred: established ssh connections or ssh connection attempts? | 22:21 |
| mordred | fungi: both | 22:21 |
| fungi | huh. packet loss? are you over ipv4 or ipv6? | 22:21 |
| mordred | ipv4 | 22:22 |
| greghaynes | fungi: I don't have the patch on hand right now from phone... clarkb? | 22:22 |
| greghaynes | (Build fail handling) | 22:23 |
| fungi | greghaynes: he got it to me, thanks | 22:23 |
| fungi | mordred: tried leaving a ping running to it and seeing if you get icmp timeouts around the same time as your ssh connection timeouts? | 22:23 |
| clarkb | ianw: +2 on the script simplification with the caveat that we may need to update it to handle pruning based on further investigation | 22:23 |
| mordred | fungi: good idea | 22:24 |
| *** Keedya_ has quit IRC | 22:24 | |
| mordred | btw - I have verified that ansible appropriately ignores the emacs autosave files | 22:24 |
| mordred | in the inventory dir | 22:24 |
| mordred | so, nevermind on the making them go away thing | 22:24 |
| fungi | mordred: for maximum win, leave another ping going to an address at the same rackspace region as a control group | 22:24 |
| clarkb | I like mtr fwiw | 22:25 |
| fungi | did it start recently? | 22:25 |
| mordred | fungi: not terribly recently, no | 22:25 |
| mordred | past few months | 22:25 |
| fungi | yeah, mtr is helpful if the loss isn't at a terminus of the connection | 22:26 |
| *** aysyd has quit IRC | 22:26 | |
| mordred | fungi: nope. timeout to puppetmaster ssh session - pings still chugging along | 22:26 |
| *** eil397 has joined #openstack-infra | 22:26 | |
| fungi | mordred: interesting, i haven't been noticing puppetmaster ssh timeouts for the past several months, though i'm also connecting mostly over ipv6 | 22:26 |
| *** ddieterly has quit IRC | 22:27 | |
| fungi | mordred: does the puppetmaster clean up the disconnected session on its end? just noticing that one of us has a couple ssh sessions connected over ipv4 for many days, so that suggests it might be something singling you out | 22:28 |
| openstackgerrit | Merged openstack-infra/project-config: Prune old branches in early devstack clone https://review.openstack.org/268841 | 22:28 |
| *** jordanP has quit IRC | 22:30 | |
| *** doug-fish has quit IRC | 22:31 | |
| Swanson | In nodepool.yaml what is the targets: section for? | 22:31 |
| clarkb | Swanson: tells nodepool what jenkins matsers to attach nodes to | 22:31 |
| *** doug-fish has joined #openstack-infra | 22:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Exclude hosts header line in expand-groups https://review.openstack.org/269312 | 22:31 |
| fungi | specifically, the jenkins master api endpoint urls | 22:31 |
| fungi | and related details | 22:32 |
| mordred | fungi, nibalizer: ^^ that patch - if'n you don't mind - just found that when the patch landed and I ran it on puppetmaster | 22:32 |
| mordred | have tested the fix | 22:32 |
| Swanson | clarkb, fungi, thanks. I must have missed it first go round and it just sez jenkins1 which has little to do with my setup. | 22:33 |
| fungi | mordred: any risk that will greedily filter things we don't want? like a host named "thishost"? (okay, silly example) | 22:33 |
| *** doug-fis_ has joined #openstack-infra | 22:34 | |
| nibalizer | mordred: what do you think about adding -m1 | 22:34 |
| nibalizer | I have similar concerns to fungi | 22:34 |
| fungi | yeah, limiting to the first match may make more sense | 22:34 |
| mordred | we could also do ^ +hosts \([0-9]+\): | 22:34 |
| fungi | that seems a lot safer | 22:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Exclude hosts header line in expand-groups https://review.openstack.org/269312 | 22:35 |
| mordred | how about that? ^^ | 22:35 |
| fungi | i feel fairly confident we'll never have a hostname matching that pattern ;) | 22:35 |
| *** doug-fish has quit IRC | 22:36 | |
| nibalizer | uh that won't work -m and -v don't combine the way I thought they did sorry | 22:36 |
| *** erlon has quit IRC | 22:36 | |
| *** pvaneck has quit IRC | 22:36 | |
| *** doug-fish has joined #openstack-infra | 22:36 | |
| nibalizer | can just drop the -m wth such a specific regex tho | 22:37 |
| clarkb | I am a little fuzzy on what those groups are for, is it purely convenience? | 22:37 |
| fungi | clarkb: conveniently applying a common role (i might not be using that word correctly) to a set of similar hosts | 22:37 |
| *** doug-fis_ has quit IRC | 22:38 | |
| openstackgerrit | Christopher Aedo proposed openstack-infra/system-config: Add PyYAML to proposal slave https://review.openstack.org/269314 | 22:38 |
| fungi | so i guess you could call that a convenience | 22:38 |
| openstackgerrit | Merged openstack/requirements: Update global-requirements.txt https://review.openstack.org/266535 | 22:38 |
| clarkb | it just seems like a lot of overhead when ansible-playbook logstash-worker* would work? | 22:38 |
| ianw | clarkb: so i think this explains what is going on -> http://paste.openstack.org/show/484212/ | 22:39 |
| *** dizquierdo has joined #openstack-infra | 22:40 | |
| clarkb | ianw: if you run git branch -a after the lsat prune you should still have a stable/juno though | 22:40 |
| clarkb | ianw: but you don't have an origin/stable/juno | 22:40 |
| ianw | clarkb: actually, the opposite? http://paste.openstack.org/show/484213/ | 22:42 |
| *** moravec has quit IRC | 22:42 | |
| *** camunoz has quit IRC | 22:42 | |
| clarkb | wah, hrm maybe I misread it when I was testing then | 22:42 |
| anteaya | fungi: did you mean to vote +2 and then remove that vote on https://review.openstack.org/#/c/269312/ ? I ask because you usually say why either on the patch or in channel if you change your vote | 22:42 |
| *** ihrachysh has quit IRC | 22:42 | |
| clarkb | ianw: in that case I think my just do a git remote update --prune may be sufficient | 22:43 |
| anteaya | so I wasn't sure if it was intentional or a gerrit derp | 22:43 |
| pabelanger | ianw: sure | 22:43 |
| clarkb | ianw: since it should update all the reachable refs and prune the invalid onces | 22:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Exclude hosts header line in expand-groups https://review.openstack.org/269312 | 22:43 |
| *** otter768 has joined #openstack-infra | 22:44 | |
| mordred | nibalizer, fungi: ^^ | 22:44 |
| *** dims has quit IRC | 22:44 | |
| keedya | hi There | 22:46 |
| keedya | How can i delete a review? | 22:46 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-gerrit: Add logic to use use available mysql module https://review.openstack.org/268349 | 22:46 |
| *** eliqiao has quit IRC | 22:46 | |
| mordred | keedya: you cannot. but you can abandon it | 22:46 |
| mordred | keedya: if you go to the review, you'll see an abandon button on it | 22:47 |
| keedya | thanks!! | 22:47 |
| *** hichihara has quit IRC | 22:47 | |
| ianw | clarkb: so actually, maybe the devstack-early clone wants to do "git fetch --prune --update-head-ok ../upstream +refs/heads/*:refs/heads/* +refs/remotes/*:refs/remotes/*" | 22:47 |
| anteaya | keedya: is this yours? https://review.openstack.org/#/c/269308/ | 22:47 |
| ianw | clarkb: is that actually an alias for what you actually said though? | 22:47 |
| *** dims has joined #openstack-infra | 22:47 | |
| ianw | yay for a million ways to do the same thing | 22:47 |
| keedya | yes this is me | 22:48 |
| *** mriedem is now known as mriedem_away | 22:48 | |
| clarkb | ianw: ya I think git remote update is basically that | 22:48 |
| keedya | i m new to the process | 22:48 |
| clarkb | ianw: but ti does it for each remote | 22:48 |
| keedya | i need to update my files | 22:48 |
| *** eliqiao has joined #openstack-infra | 22:48 | |
| anteaya | keedya: great, start here: http://docs.openstack.org/infra/manual/developers.html#updating-a-change | 22:48 |
| *** otter768 has quit IRC | 22:48 | |
| *** achanda has joined #openstack-infra | 22:48 | |
| anteaya | keedya: you prefer you update a change, rather than abandoning it and creating a new change | 22:49 |
| keedya | will do | 22:49 |
| keedya | thanks!! | 22:49 |
| anteaya | keedya: if you need help, just ask, we can then find some documentation that helps you understand our workflow | 22:49 |
| anteaya | keedya: welcome | 22:49 |
| *** achanda has quit IRC | 22:50 | |
| anteaya | s/you prefer you update a change/we prefer you update a change | 22:50 |
| keedya | anteaya i am trying to add a project written in nodejs | 22:50 |
| anteaya | yeah I saw that patch | 22:50 |
| keedya | i m having hard time finding the right documentations | 22:50 |
| anteaya | can you help me understand why you want it in nodejs? | 22:50 |
| *** weshay has quit IRC | 22:50 | |
| anteaya | most of our projects are in python | 22:50 |
| anteaya | so having a nodejs project, well we havn't seen a lot of them | 22:51 |
| anteaya | can you share some detail? | 22:51 |
| keedya | following is the code and description | 22:52 |
| keedya | https://github.com/keedya/Shovel | 22:52 |
| *** achanda has joined #openstack-infra | 22:52 | |
| *** sabeen3 has quit IRC | 22:52 | |
| anteaya | tell me the part why this has to be in nodejs and it won't function in python | 22:53 |
| *** harlowja_at_home has joined #openstack-infra | 22:55 | |
| mordred | keedya: neat - devananda have you seen that? ^^ | 22:55 |
| keedya | no major reason | 22:56 |
| *** amitgandhinz has quit IRC | 22:56 | |
| *** tonytan4ever has quit IRC | 22:56 | |
| *** thorst has quit IRC | 22:57 | |
| clarkb | the jenkins rolling restarts appear to have completed successfully last friday/saturday | 22:58 |
| *** ddieterly has joined #openstack-infra | 22:59 | |
| *** gildub has joined #openstack-infra | 22:59 | |
| *** camunoz has joined #openstack-infra | 22:59 | |
| anteaya | yay success | 23:00 |
| *** ari has quit IRC | 23:01 | |
| clarkb | nibalizer: so you have nodepool running and you are able to run d-g on the built nodes? | 23:03 |
| clarkb | nibalizer: have you managed to test the run times on >8 cpus? | 23:03 |
| *** eil397 has quit IRC | 23:04 | |
| *** bknudson has joined #openstack-infra | 23:05 | |
| *** dims has quit IRC | 23:05 | |
| ianw | clarkb: so, why would we do "git fetch --update-head-ok ../upstream/ +refs/heads/*:refs/heads/*" ... that says "get all refs/heads from remote server and put it in my refs/heads ... i.e. create local branches | 23:05 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: correct syntax error in shovel.config https://review.openstack.org/269362 | 23:05 |
| ianw | clarkb: that seems like it should be +refs/heads/*:refs/remotes/origin/* | 23:06 |
| ianw | i.e. take the remote heads, and put them under remotes/* locally | 23:06 |
| *** amotoki has quit IRC | 23:06 | |
| *** dims has joined #openstack-infra | 23:07 | |
| clarkb | ianw: hrm that is a good question, probably to avoid needing to setup tracking branches? | 23:07 |
| clarkb | make it more of a mirror instead of a clone? | 23:07 |
| mordred | keedya: so - what you'll want to do is squash those two commits into a single commit | 23:08 |
| keedya | ok | 23:08 |
| ianw | this is the type of thing a comment in the code could help with :( | 23:08 |
| mordred | keedya: gerrit tracks changes, not branches - when you squash them, make sure you throw away the Change-Id line of the second commit | 23:08 |
| mordred | keedya: that way gerrit will know to associate the new modified patch with the original patch | 23:08 |
| nibalizer | clarkb: yes nodepool works well enough | 23:09 |
| nibalizer | but every time i tried d-g last week it timed itself out at 115 minutes | 23:09 |
| nibalizer | I'll kick off another run | 23:09 |
| keedya | got it!! | 23:10 |
| *** camunoz has quit IRC | 23:10 | |
| mordred | keedya: woot! | 23:10 |
| ianw | clarkb: so you're right, the --prune in that latest commit will still not be sufficient, because the devstack image job is still going to find the refs/remotes branches and try to checkout icehouse | 23:10 |
| keedya | thanks for ur help!!! | 23:11 |
| fungi | anteaya: i removed it after nibalizer mentioned that -m1 combined with -v didn't test out after all | 23:11 |
| fungi | latest patchset lgtm though | 23:12 |
| *** baoli has joined #openstack-infra | 23:12 | |
| anteaya | fungi: ah thank you | 23:12 |
| clarkb | ianw: ya so we can do something like my naive change to iterate through and delete those that are "extra" though I think I need to update it to delete the remote branche instead of the local branches | 23:13 |
| *** achanda has quit IRC | 23:14 | |
| clarkb | I kind of assumed that git would have a better way to do that but I couldn't find one hence my brute force appraoch | 23:14 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:15 | |
| mordred | keedya: my pleasure! | 23:16 |
| *** aeng has joined #openstack-infra | 23:16 | |
| clarkb | ianw: we could look at gerrit's replication process and copy cat that. Though it does a push not pull | 23:16 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Run the ofagent ci job in the periodic-qa queue https://review.openstack.org/267323 | 23:16 |
| *** baoli has quit IRC | 23:17 | |
| *** camunoz has joined #openstack-infra | 23:22 | |
| *** achanda has joined #openstack-infra | 23:23 | |
| openstackgerrit | Merged openstack-infra/system-config: Exclude hosts header line in expand-groups https://review.openstack.org/269312 | 23:24 |
| ianw | clarkb: i wonder if "git clone --mirror" didn't exist when this was all written ... | 23:25 |
| *** rfolco has quit IRC | 23:26 | |
| clarkb | maybe? though these are not bare repos maybe they should be | 23:26 |
| clarkb | also git clone --references | 23:28 |
| clarkb | er --reference | 23:28 |
| clarkb | oh hrm that may not cross filesystems well | 23:28 |
| *** doug-fish has quit IRC | 23:28 | |
| clarkb | ya for image builds that isn't so great, however we could look into using that instead of rsync for zuul-cloner | 23:29 |
| clarkb | jeblair: ^ | 23:29 |
| ianw | so i think "git fetch --prune --update-head-ok ../upstream +refs/*:refs/*" is getting closer to what we want | 23:29 |
| ianw | and is really what a --mirror does | 23:30 |
| *** dimtruck is now known as zz_dimtruck | 23:30 | |
| clarkb | ianw: and that will delete the stale branches via the --prune? | 23:30 |
| ianw | clarkb: yes, everything (remotes too) | 23:31 |
| clarkb | beacuse we start at refs/ that makes sense | 23:31 |
| ianw | i think dib's fetch of +master:master is wrong | 23:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: You have to use egrep to get real regexes https://review.openstack.org/269365 | 23:31 |
| mordred | clarkb, fungi, nibalizer: ^^ sigh | 23:31 |
| nibalizer | lol | 23:31 |
| clarkb | ianw: refs/* includes master so it should still work for anyone relying on master | 23:32 |
| *** tsufiev has quit IRC | 23:32 | |
| *** dteselkin has quit IRC | 23:32 | |
| *** rakhmerov has quit IRC | 23:32 | |
| clarkb | ianw: but I agree master alone is quite restrictive | 23:32 |
| *** katyafervent has quit IRC | 23:33 | |
| *** salv-orlando has joined #openstack-infra | 23:33 | |
| *** harlowja_at_home has quit IRC | 23:33 | |
| jhesketh | infra-core: If anybody has time to help move the swift logs along the https://review.openstack.org/#/q/status:open+topic:enable_swift patches need reviewing (including the ones with jenkins -1 as they should work once the zuul patches are in) | 23:34 |
| *** gordc has quit IRC | 23:34 | |
| *** dizquierdo has quit IRC | 23:35 | |
| *** dims has quit IRC | 23:35 | |
| *** gildub has quit IRC | 23:35 | |
| *** pahuang has joined #openstack-infra | 23:36 | |
| *** rakhmerov has joined #openstack-infra | 23:37 | |
| *** katyafervent has joined #openstack-infra | 23:37 | |
| *** davidlenwell has quit IRC | 23:38 | |
| *** dteselkin has joined #openstack-infra | 23:38 | |
| mordred | infra-root: I self-approved the above patch because puppetmaster is otherwise broken (and it's a single character) | 23:38 |
| *** tsufiev has joined #openstack-infra | 23:39 | |
| *** dims has joined #openstack-infra | 23:39 | |
| *** dslev has quit IRC | 23:40 | |
| dougwig | mordred: no day is complete without a rage merge. | 23:40 |
| mordred | dougwig: +100 | 23:41 |
| nibalizer | mordred: kk | 23:41 |
| *** Qiming has joined #openstack-infra | 23:41 | |
| devananda | keedya: neat! | 23:42 |
| clarkb | jhesketh: the zuul changes lgtm | 23:43 |
| *** claudiub has quit IRC | 23:43 | |
| jhesketh | clarkb: ta | 23:43 |
| *** eliqiao_ has joined #openstack-infra | 23:45 | |
| ianw | clarkb: * [new ref] refs/changes/99/71199/5 -> refs/changes/99/71199/5 <-- so this is a bit of a problem with "+refs/*:refs/*" ... it gets all these which we probably don't want | 23:47 |
| clarkb | ha | 23:48 |
| clarkb | ianw: so maybe we need to do refs/heads and refs/remotes? | 23:48 |
| ianw | which is maybe why it was done like that ... and refs/remotes was just forgotten | 23:48 |
| *** pahuang has quit IRC | 23:49 | |
| *** eliqiao has quit IRC | 23:49 | |
| keedya | devanda: i m trying to add shovel to openstack CI | 23:49 |
| *** amotoki has joined #openstack-infra | 23:50 | |
| keedya | Shovel is written in nodejs | 23:50 |
| *** achanda has quit IRC | 23:50 | |
| openstackgerrit | Merged openstack-infra/system-config: You have to use egrep to get real regexes https://review.openstack.org/269365 | 23:52 |
| *** otter768 has joined #openstack-infra | 23:52 | |
| *** Swanson has left #openstack-infra | 23:52 | |
| *** pahuang has joined #openstack-infra | 23:52 | |
| *** yuanying has quit IRC | 23:54 | |
| *** rockyg has quit IRC | 23:54 | |
| mordred | infra-root: ok - the ansible group generation seems to be working now | 23:54 |
| mordred | I will doublecheck that all of the tings are being copied where we expect all the things to be copied to tomorrow | 23:55 |
| *** rajinir has quit IRC | 23:55 | |
| mordred | and if that is happening, I plan on getting medevial on finishing up puppet-apply | 23:55 |
| nibalizer | woot | 23:55 |
| mordred | because it's on my last nerve | 23:55 |
| *** yuanying has joined #openstack-infra | 23:56 | |
| *** gildub has joined #openstack-infra | 23:57 | |
| *** davidlenwell has joined #openstack-infra | 23:58 | |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Perform filepath checking in python https://review.openstack.org/266439 | 23:58 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Collapse redundant hieradata tasks https://review.openstack.org/265652 | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
