| pmoosh | Hello guys - a newbie question | 00:04 |
|---|---|---|
| *** hashar has quit IRC | 00:04 | |
| clarkb | fungi: your stack is +2 from me | 00:05 |
| *** dims has joined #openstack-infra | 00:05 | |
| clarkb | fungi: I think we can worry about auto updates in fancier ways later if you like | 00:06 |
| clarkb | pmoosh: go for it | 00:06 |
| pmoosh | the last Jenkins died with a timeout | 00:06 |
| pmoosh | so what do I do then | 00:07 |
| clarkb | pmoosh: can you link to the change/test? | 00:07 |
| pmoosh | here is the log: http://logs.openstack.org/25/46325/1/check/gate-requirements-install/0894d3a | 00:08 |
| pmoosh | 46325 - think problem is Error The read operation timed out while getting https://pypi.python.org/packages/source/p/python-heatclient/python-heatclient-0.2.4.tar.gz#m | 00:09 |
| clarkb | pmoosh: that is a timeout when talking to pypi.python.org. Typically we use our mirror instead of pypi.python.org, but this job tests the addition of new things to the mirror so we can't rely on our mirror here | 00:10 |
| clarkb | pmoosh: for now you will probably need to recheck the test | 00:10 |
| clarkb | mordred: ^ any opinion on how we should handle this in the future? | 00:10 |
| *** prad has quit IRC | 00:12 | |
| pmoosh | thanks - how would I do that - I guess that is the newbie question :-) | 00:12 |
| clarkb | pmoosh: in the comment that posts the results is a link to a wiki page describing what you should do on failures | 00:12 |
| clarkb | pmoosh: in this case leave a comment of 'recheck no bug' on the change | 00:12 |
| clarkb | that will retrigger the jobs. We could submit a bug and recheck against that, but I am not sure it is worth tracking 'pypi timed out' | 00:13 |
| anteaya | fungi: do you want to see if we can round up more votes and roll in the salt patches tonight (let them run all night and evaluate memory leaks with Dave when he is around?) or would you rather wait? | 00:14 |
| *** prad has joined #openstack-infra | 00:14 | |
| *** changbl has joined #openstack-infra | 00:14 | |
| anteaya | fungi: if there are any memory leaks (which I hope there aren't) | 00:15 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use Ubuntu Cloud Archive on precise nodes https://review.openstack.org/45896 | 00:17 |
| openstackgerrit | A change was merged to openstack-infra/publications: Make make-index ignore template branch. https://review.openstack.org/46356 | 00:18 |
| *** pmoosh has quit IRC | 00:23 | |
| clarkb | annegentle: are you guys ready to merge https://review.openstack.org/#/c/45938/1 ? | 00:24 |
| clarkb | annegentle: the change noted in the commit message merged | 00:24 |
| jeblair | clarkb, fungi: fyi: i'm on a plane for most of tomorrow | 00:34 |
| clarkb | jeblair: ok any chance you can review https://review.openstack.org/#/c/41927/ before your flight? | 00:35 |
| clarkb | jeblair: curious to see what you think about fungi's comment in particular | 00:35 |
| jeblair | clarkb, mordred: well, i do wonder why one line of shell is in JJB instead of the shell script... | 00:39 |
| *** pcrews has joined #openstack-infra | 00:40 | |
| *** mgagne1 is now known as mgagne | 00:43 | |
| *** reed has quit IRC | 00:47 | |
| *** yongli has joined #openstack-infra | 00:48 | |
| *** pcrews has quit IRC | 00:49 | |
| clarkb | jeblair: https://review.openstack.org/#/c/46125/ what do you think of my comment there? | 00:52 |
| *** HenryG has joined #openstack-infra | 00:59 | |
| *** branen_ has joined #openstack-infra | 01:00 | |
| *** jog0 has quit IRC | 01:00 | |
| *** jog0 has joined #openstack-infra | 01:01 | |
| *** branen has quit IRC | 01:03 | |
| *** gyee has quit IRC | 01:04 | |
| annegentle | clarkb: I'm good with https://review.openstack.org/#/c/45938/1 | 01:05 |
| clarkb | annegentle: done | 01:06 |
| annegentle | clarkb: thanks! | 01:06 |
| *** dkranz has joined #openstack-infra | 01:06 | |
| *** nosnos has joined #openstack-infra | 01:06 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move admin-guide-cloud to docs.o.o/admin-guide-cloud/content https://review.openstack.org/45938 | 01:07 |
| mrodden | where is the best place for jenkins-job-builder IRC discussion... here? | 01:08 |
| fungi | anteaya: i'm mostly transitioning to afk status for the evening, but the previous memory/process leak took weeks to manifest anyway | 01:08 |
| fungi | anteaya: i think i should be able to drive this the rest of the way home. UtahDave's pointers have been most helpful | 01:10 |
| *** svarnau has joined #openstack-infra | 01:11 | |
| anteaya | fungi: happy evening then | 01:13 |
| anteaya | so glad it has been a good week for Salt | 01:13 |
| *** svarnau_ has joined #openstack-infra | 01:13 | |
| anteaya | I should afk too | 01:13 |
| clarkb | mrodden: yes here is the best place | 01:13 |
| anteaya | I'm sure I have laundry/packing to do | 01:13 |
| mrodden | so i noticed there was a lack of a git publisher plugin | 01:14 |
| mrodden | and i kind of need that for my job, so would you guys accept plugins? | 01:14 |
| *** oubiwann has quit IRC | 01:15 | |
| mrodden | i have to write it anyways | 01:15 |
| clarkb | mrodden: yes, we accept patches that support more plugins | 01:15 |
| mrodden | k cool | 01:15 |
| fungi | in fact, most of the contributed patches are plugin support | 01:15 |
| fungi | pick a random jjb commit and it's likely adding support for another plugin | 01:17 |
| mrodden | i was surprised there wasn't a Git Publisher one | 01:17 |
| fungi | seems it was waiting for you ;) | 01:18 |
| clarkb | mrodden: pushing to git from jenkins is something we avoid | 01:18 |
| clarkb | mrodden: and when we need it, we have git review which works really well without plugin support | 01:18 |
| mrodden | how does Jenkins merge patches at gate? | 01:18 |
| clarkb | mrodden: it doesn't, instead gerrit does it for us | 01:19 |
| *** svarnau_ has quit IRC | 01:19 | |
| fungi | rather, zuul tells gerrit to merge at the moment | 01:19 |
| mrodden | so... jenkins just invokes the Gerrit submit action? | 01:19 |
| mrodden | oh ok | 01:19 |
| *** markmcclain has quit IRC | 01:19 | |
| *** Transformer has joined #openstack-infra | 01:19 | |
| clarkb | mrodden: technically zuul does that but yes | 01:19 |
| *** svarnau has quit IRC | 01:20 | |
| mrodden | i was originally really nervous about having jenkins pushing to git | 01:21 |
| fungi | in the future zuul may gain the ability to merge commits itself, but the details have yet to be worked out entirely, i think | 01:21 |
| mrodden | but i've been running some jobs that grab upstream code and merge some internal stuff, then run all the unit tests and push to a downstream repo if successful | 01:21 |
| *** Transformer has quit IRC | 01:21 | |
| mrodden | its saved me an immeasurable amount of time | 01:22 |
| fungi | neat! | 01:22 |
| mrodden | yeah i used to do all that by hand every week | 01:22 |
| mrodden | now its daily and automated :) | 01:22 |
| fungi | and yeah, jjb has grown support for all sorts of plugins we don't use for openstack infra. perfectly fine | 01:23 |
| mrodden | yeah i was impressed by the amount of plugin support | 01:23 |
| mrodden | i've learned so much working on openstack stuff its crazy | 01:25 |
| mrodden | like | 01:25 |
| mrodden | python setup.py develop is the best thing ever | 01:25 |
| *** dcramer__ has joined #openstack-infra | 01:26 | |
| *** nati_ueno has quit IRC | 01:27 | |
| mrodden | plus there is all the great tools that are in openstack-infra | 01:28 |
| *** jhesketh has quit IRC | 01:45 | |
| openstackgerrit | Giulio Fidente proposed a change to openstack-infra/devstack-gate: extend the floating ips range for neutron tests https://review.openstack.org/46403 | 01:48 |
| *** cody-somerville has quit IRC | 01:50 | |
| *** xchu has joined #openstack-infra | 01:50 | |
| *** cody-somerville has joined #openstack-infra | 01:53 | |
| *** cody-somerville has quit IRC | 01:53 | |
| *** cody-somerville has joined #openstack-infra | 01:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix paths for glossary https://review.openstack.org/46326 | 01:54 |
| *** nhm has quit IRC | 01:55 | |
| *** dzyu has joined #openstack-infra | 01:56 | |
| *** zoresvit has quit IRC | 01:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix publishing of High Availibility Guide https://review.openstack.org/46333 | 02:00 |
| *** dzyu has quit IRC | 02:03 | |
| *** dzyu has joined #openstack-infra | 02:06 | |
| *** melwitt has quit IRC | 02:12 | |
| *** yaguang has joined #openstack-infra | 02:12 | |
| openstackgerrit | Giulio Fidente proposed a change to openstack-infra/devstack-gate: extend the floating ips range (for neutron tests to pass) https://review.openstack.org/46403 | 02:14 |
| *** anteaya has quit IRC | 02:32 | |
| *** cyeoh has quit IRC | 02:36 | |
| *** dzyu has quit IRC | 02:43 | |
| *** dzyu has joined #openstack-infra | 02:43 | |
| *** dcramer__ has quit IRC | 02:48 | |
| *** dims has quit IRC | 03:08 | |
| *** mgagne has quit IRC | 03:26 | |
| mordred | mrodden: whee! | 03:33 |
| *** nicedice has quit IRC | 03:34 | |
| *** nicedice has joined #openstack-infra | 03:34 | |
| *** mgagne has joined #openstack-infra | 03:43 | |
| *** mgagne has joined #openstack-infra | 03:43 | |
| *** mgagne1 has joined #openstack-infra | 03:45 | |
| *** mgagne1 has joined #openstack-infra | 03:45 | |
| *** mgagne has quit IRC | 03:48 | |
| mordred | clarkb: re just recheck when that happens (the requirements thing) | 03:49 |
| mordred | clarkb: the idea is that changes to requirements shoudl not be happening so often that it's that much of a burden to recheck on pypi derp | 03:49 |
| *** vogxn has joined #openstack-infra | 03:57 | |
| *** mgagne1 has quit IRC | 03:57 | |
| *** xchu has quit IRC | 04:01 | |
| Alex_Gaynor | dhellmann: in case you missed it, the patch to cmd2 was merged | 04:10 |
| *** xchu has joined #openstack-infra | 04:14 | |
| *** stevebaker has quit IRC | 04:19 | |
| *** adam_g has quit IRC | 04:19 | |
| *** torgomatic has quit IRC | 04:19 | |
| *** russellb has quit IRC | 04:19 | |
| *** russellb has joined #openstack-infra | 04:22 | |
| *** stevebaker has joined #openstack-infra | 04:22 | |
| mordred | woot | 04:23 |
| * mordred forgets why we wanted that | 04:23 | |
| *** torgomatic has joined #openstack-infra | 04:23 | |
| mordred | btw - /me needs to finisht eh github PR trigger for zuul so that we can start telling upstream projects we use when their patches are going to break us | 04:24 |
| * mordred is SURE that they all care massively | 04:24 | |
| *** oubiwann has joined #openstack-infra | 04:24 | |
| *** cyeoh_ has joined #openstack-infra | 04:25 | |
| *** cyeoh_ is now known as cyeoh | 04:25 | |
| *** jhesketh__ has quit IRC | 04:33 | |
| lifeless | mordred: see my ping earlier ? | 04:36 |
| lifeless | mordred: requirements.txt files with URLs in them break dib offline mode from being well, offline. | 04:37 |
| mordred | lifeless: yup | 04:38 |
| mordred | I mean - no, I did not see the ping, and yes, files with urls in them break dib offline mode | 04:38 |
| mordred | currently the only urls should be for oslo pre-releases | 04:38 |
| mordred | I'm working on the infrastructure needed to make those go away | 04:39 |
| mordred | (the urls for the pre-releases) | 04:39 |
| mordred | once I do, I'd like to ban requirement urls | 04:39 |
| *** jhesketh__ has joined #openstack-infra | 04:46 | |
| lifeless | cool | 04:48 |
| *** SergeyLukjanov has joined #openstack-infra | 04:49 | |
| mordred | lifeless: this needs the wheel in mirror support to land, which I think it getting closer | 04:51 |
| *** vogxn has quit IRC | 04:53 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/cookiecutter: Fix the README url location for the repo https://review.openstack.org/46414 | 04:59 |
| openstackgerrit | A change was merged to openstack-dev/cookiecutter: Fix the README url location for the repo https://review.openstack.org/46414 | 04:59 |
| *** yaguang has quit IRC | 05:03 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/cookiecutter: Add pypy to the list of default envs https://review.openstack.org/46416 | 05:06 |
| *** SergeyLukjanov has quit IRC | 05:08 | |
| Alex_Gaynor | mordred: nice stuff with cookie cutter! | 05:13 |
| mordred | Alex_Gaynor: thanks! | 05:13 |
| *** pcrews has joined #openstack-infra | 05:13 | |
| *** dmakogon_ has joined #openstack-infra | 05:14 | |
| *** vogxn has joined #openstack-infra | 05:15 | |
| openstackgerrit | A change was merged to openstack/requirements: Remove version pins from setup_requires https://review.openstack.org/45311 | 05:17 |
| openstackgerrit | Mathew Odden proposed a change to openstack-infra/jenkins-job-builder: Add publisher for Git Publisher support https://review.openstack.org/46417 | 05:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Minify jquery visibility https://review.openstack.org/46335 | 05:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Delete closed bugs after 5 days in recheckwatch https://review.openstack.org/46169 | 05:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Lower devstack-gate timeouts https://review.openstack.org/46154 | 05:22 |
| openstackgerrit | A change was merged to openstack-infra/config: supress maven download progress messages https://review.openstack.org/46142 | 05:23 |
| openstackgerrit | A change was merged to openstack-infra/config: Move running-your-own to the sidebar https://review.openstack.org/46131 | 05:24 |
| Alex_Gaynor | mordred: I take it https://github.com/openstack-dev/cookiecutter is the official location for this, not the one on your account right? | 05:25 |
| mordred | Alex_Gaynor: yes. that's right | 05:25 |
| mordred | Alex_Gaynor: the one on my account isn't referenced in a doc anywhere is it? | 05:26 |
| Alex_Gaynor | mordred: it's in the cookiecutter readme, I'm sending a PR to fix now | 05:26 |
| mordred | thanks | 05:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Add pre-release PyPI jobs for WSME https://review.openstack.org/46106 | 05:26 |
| Alex_Gaynor | mordred: yay, opensource https://github.com/audreyr/cookiecutter/pull/72 | 05:27 |
| mordred | Alex_Gaynor: oh wait- do you mean https://github.com/audreyr/cookiecutter. ? | 05:27 |
| mordred | Alex_Gaynor: AH - neat | 05:27 |
| *** nicedice has quit IRC | 05:27 | |
| mordred | btw - docopt seems really neat | 05:30 |
| * Alex_Gaynor has mixed opinions on docopt | 05:30 | |
| mordred | I just saw it referenced on that cookiecutter page, so have had very low amounts of time to think about it | 05:31 |
| mordred | what I REALLY wnat python to grow | 05:32 |
| mordred | is a thing that works like boost's program_options | 05:32 |
| *** yaguang has joined #openstack-infra | 05:32 | |
| mordred | http://www.boost.org/doc/libs/1_54_0/doc/html/program_options.html | 05:32 |
| mordred | which is not a thing I'm going to say every day | 05:32 |
| *** michchap has joined #openstack-infra | 05:32 | |
| Alex_Gaynor | I've spent about 12 seconds looking at this, but how is it significantly different from argparse? | 05:33 |
| mordred | but grokking that what you want to do is describe the things you want to configure, and then having all of those immediately be consistently settable via env vars, config files and command line invocation is nice | 05:33 |
| mordred | Alex_Gaynor: arg parse only does command line options | 05:33 |
| Alex_Gaynor | Ah, this handles config files as well? | 05:33 |
| mordred | yup. and env vars | 05:33 |
| mordred | but it handles it pretty much how argparse handles args | 05:34 |
| mordred | so, it works pretty much exactly the same as argparse | 05:34 |
| mordred | you just get the other two as well | 05:34 |
| Alex_Gaynor | right, makes sense | 05:34 |
| Alex_Gaynor | I feel like maybe I heard about a thing that does this somewhere. | 05:34 |
| *** xchu has quit IRC | 05:35 | |
| mordred | well, oslo.config does it - but does it with those global config objects | 05:35 |
| mordred | (unfortunately, openstack inherited all of this gflags nonsense) | 05:35 |
| Alex_Gaynor | ah, is there where it comes from | 05:36 |
| mordred | yeah | 05:37 |
| mordred | iirc, that's what nova was using when we started | 05:38 |
| mordred | yup | 05:39 |
| mordred | it's in commit 1 | 05:39 |
| mordred | of course, commit 1 _also_ includes vendor dirs for carrot, pymox, redis-py, tornado and Twisted-10.0.0 | 05:40 |
| Alex_Gaynor | were they using twisted and carrot together? | 05:40 |
| * Alex_Gaynor focuses on the important details | 05:41 | |
| mordred | Alex_Gaynor: yes | 05:41 |
| mordred | from carrot import connection | 05:41 |
| mordred | from carrot import messaging | 05:41 |
| mordred | from twisted.internet import defer | 05:41 |
| mordred | from twisted.internet import reactor | 05:41 |
| mordred | from twisted.internet import task | 05:41 |
| Alex_Gaynor | neat | 05:42 |
| *** Ajaeger has joined #openstack-infra | 05:42 | |
| mordred | wow. commit 1 is so small - if you ignore the vendor dir | 05:44 |
| openstackgerrit | A change was merged to openstack/requirements: Add a list of projects that we want to sync to https://review.openstack.org/46347 | 05:45 |
| *** pblaho has joined #openstack-infra | 05:47 | |
| *** xchu has joined #openstack-infra | 05:47 | |
| Ajaeger | Good morning. | 05:48 |
| *** Ryan_Lane has joined #openstack-infra | 05:59 | |
| *** jhesketh__ has quit IRC | 06:13 | |
| *** jpeeler has quit IRC | 06:18 | |
| *** nayward has joined #openstack-infra | 06:18 | |
| *** yolanda has joined #openstack-infra | 06:21 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix publish location of High Availibility Guide https://review.openstack.org/46425 | 06:23 |
| *** flaper87|afk is now known as flaper87 | 06:39 | |
| *** vogxn has quit IRC | 06:43 | |
| *** vogxn has joined #openstack-infra | 06:46 | |
| *** dmakogon_ has quit IRC | 06:48 | |
| *** sld-away is now known as sld | 06:48 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use first template for manual build jobs https://review.openstack.org/45574 | 06:52 |
| *** yaguang has quit IRC | 06:53 | |
| Ajaeger | jeblair: ^^^ Ready to merge - if you like it ;) | 06:55 |
| *** yaguang has joined #openstack-infra | 06:55 | |
| *** yaguang has quit IRC | 06:55 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use first template for manual build jobs https://review.openstack.org/45574 | 06:57 |
| *** yaguang has joined #openstack-infra | 06:58 | |
| *** yaguang has quit IRC | 07:08 | |
| *** dizquierdo has joined #openstack-infra | 07:11 | |
| *** yaguang has joined #openstack-infra | 07:11 | |
| *** michchap has quit IRC | 07:15 | |
| *** yaguang has quit IRC | 07:19 | |
| *** vogxn has quit IRC | 07:30 | |
| *** HenryG has quit IRC | 07:30 | |
| *** HenryG has joined #openstack-infra | 07:31 | |
| *** afazekas has joined #openstack-infra | 07:31 | |
| openstackgerrit | YangLei proposed a change to openstack/requirements: Update the version of cliff in global-requirements.txt https://review.openstack.org/46439 | 07:33 |
| *** yaguang has joined #openstack-infra | 07:38 | |
| *** turul_ has joined #openstack-infra | 07:44 | |
| *** thomasbiege has joined #openstack-infra | 07:44 | |
| *** afazekas is now known as afazekas__ | 07:44 | |
| *** turul_ is now known as afazekas | 07:44 | |
| *** salv-orlando has quit IRC | 07:50 | |
| *** yaguang has quit IRC | 07:52 | |
| *** nayward has quit IRC | 07:52 | |
| *** thomasbiege has quit IRC | 07:59 | |
| *** fbo_away is now known as fbo | 08:05 | |
| *** dzyu has quit IRC | 08:05 | |
| *** nayward has joined #openstack-infra | 08:06 | |
| *** kurt_jiang has joined #openstack-infra | 08:07 | |
| *** jpich has joined #openstack-infra | 08:07 | |
| *** thomasbiege has joined #openstack-infra | 08:09 | |
| openstackgerrit | Yaguang Tang proposed a change to openstack/requirements: Update boto not use the version 2.13.0 https://review.openstack.org/46449 | 08:10 |
| *** yaguang has joined #openstack-infra | 08:11 | |
| *** ndipanov has joined #openstack-infra | 08:19 | |
| ndipanov | hey guys - any core requirements people here? we need this to unbreak nova gate it seems https://review.openstack.org/#/c/46449/ | 08:20 |
| *** yaguang has quit IRC | 08:20 | |
| *** yaguang has joined #openstack-infra | 08:22 | |
| *** thomasbiege has quit IRC | 08:23 | |
| *** salv-orlando has joined #openstack-infra | 08:24 | |
| *** dzyu has joined #openstack-infra | 08:37 | |
| *** afazekas has quit IRC | 08:38 | |
| ndipanov | jd__, ping | 08:40 |
| *** dzyu has quit IRC | 08:41 | |
| *** afazekas__ has quit IRC | 08:42 | |
| *** ianw has quit IRC | 08:47 | |
| *** dizquierdo has left #openstack-infra | 08:59 | |
| *** adam_g has joined #openstack-infra | 09:15 | |
| *** adam_g has quit IRC | 09:15 | |
| *** adam_g has joined #openstack-infra | 09:15 | |
| *** ndipanov has quit IRC | 09:17 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Build Blocker Plugin support added https://review.openstack.org/46262 | 09:34 |
| *** yongli has quit IRC | 09:34 | |
| *** alexpilotti has joined #openstack-infra | 09:36 | |
| *** boris-42 has joined #openstack-infra | 09:36 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:41 | |
| *** afazekas__ has joined #openstack-infra | 09:42 | |
| *** SergeyLukjanov has quit IRC | 09:54 | |
| *** xchu has quit IRC | 09:58 | |
| *** odyssey4me has joined #openstack-infra | 09:58 | |
| *** alexpilotti has quit IRC | 10:04 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:10 | |
| *** koobs has quit IRC | 10:20 | |
| *** nosnos has quit IRC | 10:26 | |
| *** nosnos has joined #openstack-infra | 10:27 | |
| *** nosnos has quit IRC | 10:27 | |
| *** nosnos has joined #openstack-infra | 10:28 | |
| *** nayward has quit IRC | 10:29 | |
| *** vogxn has joined #openstack-infra | 10:35 | |
| *** nosnos has quit IRC | 10:35 | |
| *** thomasbiege has joined #openstack-infra | 10:35 | |
| *** nosnos has joined #openstack-infra | 10:36 | |
| *** nosnos has quit IRC | 10:40 | |
| *** pcm_ has joined #openstack-infra | 10:41 | |
| *** pcm_ has quit IRC | 10:43 | |
| *** pcm_ has joined #openstack-infra | 10:44 | |
| SergeyLukjanov | mordred, https://review.openstack.org/#/c/46347/ is it needed to automatically create change requests with requirements updates? | 10:52 |
| *** ruhe has joined #openstack-infra | 10:57 | |
| *** yaguang has quit IRC | 10:59 | |
| *** vogxn has quit IRC | 11:07 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Build Blocker Plugin support added https://review.openstack.org/46262 | 11:08 |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Build Blocker Plugin support added https://review.openstack.org/46262 | 11:13 |
| *** afazekas__ is now known as afazekas_food | 11:16 | |
| *** ruhe has quit IRC | 11:16 | |
| *** kurt_jiang has left #openstack-infra | 11:17 | |
| *** alexpilotti has joined #openstack-infra | 11:19 | |
| *** ruhe has joined #openstack-infra | 11:19 | |
| *** thomasbiege2 has joined #openstack-infra | 11:21 | |
| *** thomasbiege has quit IRC | 11:24 | |
| *** afazekas_food is now known as afazekas | 11:29 | |
| *** thomasbiege2 has quit IRC | 11:35 | |
| *** dizquierdo has joined #openstack-infra | 11:36 | |
| *** nayward has joined #openstack-infra | 11:44 | |
| *** adalbas has joined #openstack-infra | 11:52 | |
| openstackgerrit | Paul Bourke proposed a change to openstack/requirements: Add ordereddict for python 2.6 compatibility https://review.openstack.org/46474 | 11:53 |
| *** nayward_ has joined #openstack-infra | 12:03 | |
| *** weshay has joined #openstack-infra | 12:03 | |
| *** nayward has quit IRC | 12:04 | |
| *** thomasm has joined #openstack-infra | 12:07 | |
| *** yamahata has quit IRC | 12:07 | |
| *** yamahata has joined #openstack-infra | 12:07 | |
| *** dizquierdo has quit IRC | 12:17 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Ignore hacking warnings https://review.openstack.org/46481 | 12:20 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Build a images using diskimage-builder https://review.openstack.org/46482 | 12:20 |
| mordred | SergeyLukjanov: I think it will be helpful to - any update to global requirements _does_ want to be syncced with every project | 12:21 |
| mordred | SergeyLukjanov: and it's automatable - so why not have scripts do it | 12:22 |
| *** dizquierdo has joined #openstack-infra | 12:22 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 12:23 |
| SergeyLukjanov | mordred, ok thx | 12:23 |
| *** stevebaker has quit IRC | 12:24 | |
| *** stevebaker has joined #openstack-infra | 12:24 | |
| *** michchap has joined #openstack-infra | 12:25 | |
| *** rfolco has joined #openstack-infra | 12:26 | |
| SergeyLukjanov | are there thoughts about https://review.openstack.org/#/c/37620 ? | 12:26 |
| *** SergeyLukjanov has quit IRC | 12:29 | |
| *** zeus has joined #openstack-infra | 12:30 | |
| *** dizquierdo has quit IRC | 12:31 | |
| mordred | looking | 12:32 |
| mordred | sdague: morning | 12:33 |
| *** dprince has joined #openstack-infra | 12:33 | |
| *** CaptTofu has quit IRC | 12:34 | |
| *** zeus has quit IRC | 12:34 | |
| *** ArxCruz has joined #openstack-infra | 12:35 | |
| *** michchap has quit IRC | 12:38 | |
| *** michchap has joined #openstack-infra | 12:39 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:42 | |
| *** erfanian has quit IRC | 12:44 | |
| Ajaeger | mordred: regarding the manual templates: Yes, we need them. Fixing them one by one I learn what a mess they are ;) | 12:44 |
| *** zul has quit IRC | 12:48 | |
| *** bingbu has joined #openstack-infra | 12:49 | |
| *** erfanian has joined #openstack-infra | 12:54 | |
| *** matty_dubs|gone is now known as matty_dubs | 12:56 | |
| *** zul has joined #openstack-infra | 12:57 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/reviewday: Update bin/reviewday to use neutron. https://review.openstack.org/46487 | 12:58 |
| *** prad has quit IRC | 12:58 | |
| *** jcoufal has joined #openstack-infra | 13:02 | |
| *** bingbu has quit IRC | 13:03 | |
| *** KLevenstein has joined #openstack-infra | 13:04 | |
| *** anteaya has joined #openstack-infra | 13:07 | |
| *** michchap has quit IRC | 13:08 | |
| *** nayward_ has quit IRC | 13:09 | |
| *** ruhe has quit IRC | 13:13 | |
| *** nayward has joined #openstack-infra | 13:25 | |
| dhellmann | Alex_Gaynor, mordred: the upstream cmd2 change was +1 by someone, but I still see the PR open. did you mean something else? | 13:25 |
| *** ruhe has joined #openstack-infra | 13:28 | |
| *** SergeyLukjanov has quit IRC | 13:32 | |
| *** nayward has quit IRC | 13:33 | |
| *** jpeeler has joined #openstack-infra | 13:34 | |
| *** jpeeler has joined #openstack-infra | 13:34 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Move requirements install to a script https://review.openstack.org/46495 | 13:37 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Update projects when requirements change https://review.openstack.org/40068 | 13:37 |
| mordred | dhellmann: me have no clue | 13:37 |
| mordred | dhellmann: btw - I'm in your state for about 30 minutes :) | 13:37 |
| dhellmann | mordred: aw, next time call first :-) | 13:38 |
| dhellmann | mordred: regarding the cmd2 thing, Alex_Gaynor said something merged last night | 13:39 |
| dhellmann | mordred: seemed to be part of a conversation with you, but maybe not | 13:39 |
| mordred | dhellmann: I just also saw that he said that | 13:39 |
| mordred | and I was supportive | 13:39 |
| fungi | mordred: layover in atl huh? | 13:40 |
| mordred | fungi: you know it | 13:40 |
| dhellmann | mordred: should still be early enough for a chicken biscuit :-) | 13:40 |
| * fungi loves walking between the a and e concourses in atl | 13:40 | |
| fungi | that place is some great exercise | 13:41 |
| mordred | dhellmann: I am sadly sitting in my next seat already | 13:41 |
| mordred | fungi: ++ | 13:41 |
| mordred | fungi: it's even more fun when you need to get between them very quickly :) | 13:41 |
| fungi | yup | 13:41 |
| fungi | with bags even ;) | 13:42 |
| dhellmann | once, long ago, before the TSA, I ran from the ticket agent to terminal e and right onto a plane just before the doors closed | 13:42 |
| mordred | those were the days! | 13:42 |
| dhellmann | I am still panting. | 13:42 |
| mordred | olaph did that once years ago - except his was on the tail end of a mad drive from durham to dc | 13:43 |
| mordred | basically, pull up, run in airport, run on plane. DONE | 13:43 |
| mordred | (I can do this again, now that I have the TSA precheck) | 13:43 |
| *** prad has joined #openstack-infra | 13:44 | |
| mordred | of course, I've heard that they're opening the floodgates on letting people get that, so I'm guessing I'll have an airport line again at some point | 13:44 |
| dhellmann | they're starting their own pay-to-play thing, comparable to global entry | 13:44 |
| *** Ajaeger1 has joined #openstack-infra | 13:44 | |
| *** Ajaeger has quit IRC | 13:44 | |
| dhellmann | but just for domestic flights | 13:44 |
| mordred | yup | 13:44 |
| * fungi had a similar experience with a transatlantic flight from clt to lgw... had to get a replacement passport issued while on a 2.5-hour layover in phl... cab from the airport to the national passport office, photos at a joint around the corner, printed right there and cab back to the airport in time the next leg of my flight | 13:45 | |
| mordred | sandy just got global entry so that when she travels with me I don't get annoyed at her making me stand in lines | 13:45 |
| fungi | would never happen these days | 13:45 |
| mordred | fungi: wow. impressive! | 13:45 |
| fungi | i did not think it possible even then | 13:45 |
| fungi | was sure i'd wind up missing the flight and spending the night there | 13:45 |
| *** nayward has joined #openstack-infra | 13:46 | |
| mordred | I was flying once with my friend talia to her mom's place on the island of jersey. we had a 1 hour stopover on guernsey | 13:46 |
| fungi | ha | 13:46 |
| mordred | during which her friend met us at the airport, we drove to her friend's house, had tea, petted the cats, and then drove back to the airport to board the plane | 13:46 |
| anteaya | ++ layover with cats | 13:46 |
| fungi | oh, wow. so an hour really is enough time to see guernsey. i always suspected | 13:46 |
| mordred | yup | 13:47 |
| mordred | and all layovers should really have cats | 13:47 |
| mordred | jeblair: when you awaken, you may notice that I put up a review for dib + nodepool | 13:47 |
| mordred | jeblair: I'm not convinced that nodepool is the right place fortthe elements | 13:48 |
| mordred | so I did not add code to actually run dib to create the image locally | 13:48 |
| mordred | but I did add code to upload an image and run the takeover node stuff | 13:48 |
| *** tstevenson has joined #openstack-infra | 13:48 | |
| mordred | I figure chatting about when/where the image building should happen is in order | 13:49 |
| * dhellmann considers starting a chain of in-airport relaxation centers with cats for petting | 13:49 | |
| mordred | also, the heat guys are working on using dib to make images for use in devstack - and we had a brief conversation about post jobs to build the appropriate image and save it somwhere so that most devstack runs can just consume that | 13:50 |
| mordred | dhellmann: you would make a fortune | 13:50 |
| mordred | ok. taking off. biab | 13:50 |
| ArxCruz | ALL: I'm having problem to run puppet in my local server, it's throwing this error: http://paste.openstack.org/show/47003/ | 13:52 |
| *** frankdeez has joined #openstack-infra | 13:52 | |
| frankdeez | hello can anyone see this? | 13:52 |
| fungi | ArxCruz: it looks like you're missing a definition in the calling class for the memorytotalbytes variable | 13:53 |
| *** thedodd has joined #openstack-infra | 13:53 | |
| fungi | frankdeez: yes, i can | 13:53 |
| frankdeez | ok great | 13:53 |
| fungi | welcome to the exciting world of irc | 13:53 |
| ArxCruz | fungi: yes, actualyy, the memorytotalbytes have a sed command that seems it's not working | 13:53 |
| frankdeez | is there a text-only link to the OpenStack Corporate CLA? i need to send to my boss for review | 13:54 |
| frankdeez | sort of like this but the Corporate version: https://review.openstack.org/static/cla.html | 13:54 |
| fungi | ArxCruz: ahh, yes it looks like maybe it gets defined by modules/jenkins/lib/facter/memorytotalbytes.rb | 13:55 |
| ArxCruz | fungi: exactly | 13:55 |
| ArxCruz | fungi: do you know how can I fix that ? | 13:55 |
| ArxCruz | Facter::Util::Resolution.exec('free -bo | sed -n \'s/^Mem:\W\+\([0-9]\+\).*$/\1/p\'') | 13:55 |
| fungi | frankdeez: reed would know, but he doesn't seem to be around at the moment. you might try e-mailing communitymngr@openstack.org | 13:56 |
| frankdeez | ok super thx! | 13:57 |
| fungi | frankdeez: we don't have any automation built around the corp and usg contribution agreements since they tend to result in discussions between lawyers for both sides often enough to not benefit from that anyway | 13:58 |
| fungi | ArxCruz: digging into it to see what i can figure out | 13:58 |
| ekarlso- | how do I do if I formerly signed a individual CLA and now need a Corp ? | 13:59 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/jeepyb: Alter comments left on 'Related' bugs https://review.openstack.org/46499 | 13:59 |
| ArxCruz | fungi: if I execute te command manually I got this error: | 13:59 |
| ArxCruz | $ free -bo | sed -n \'s/^Mem:\W\+\([0-9]\+\).*$/\1/p\' | 13:59 |
| ArxCruz | sed: -e expression #1, char 1: unknown command: `'' | 13:59 |
| fungi | ArxCruz: does free -bo | sed -n \'s/^Mem:\W\+\([0-9]\+\).*$/\1/p\' return a sane number on that system? | 13:59 |
| fungi | ahh | 14:00 |
| fungi | right, no need to escape the ' marks when running int yourself | 14:00 |
| fungi | free -bo | sed -n 's/^Mem:\W\+\([0-9]\+\).*$/\1/p' | 14:00 |
| fungi | 264929280 | 14:00 |
| ArxCruz | $ free -bo | sed -n 's/^Mem:\W\+\([0-9]\+\).*$/\1/p' | 14:00 |
| ArxCruz | 2101010432 | 14:00 |
| fungi | okay, so it should be returning that value back for the "memorytotalbytes" fact | 14:01 |
| ArxCruz | yup, but it's not :/ | 14:01 |
| ArxCruz | actually, it's returning nothing, so I believe it's not getting the memorytotalbytes from modules/jenkins/lib/facter/memorytotalbytes.rb | 14:01 |
| *** mriedem has joined #openstack-infra | 14:02 | |
| Alex_Gaynor | dhellmann: anything I should knwo before I approve https://review.openstack.org/#/c/46439/1? | 14:02 |
| fungi | yeah, when i run 'facter memorytotalbytes' on random jenkins slaves of ours, i see i get a null string back | 14:02 |
| fungi | ArxCruz: however that's also apparently facter's behavior when you pass it the name of a fact it doesn't know about | 14:03 |
| fungi | so it could be that it's not finding/running that | 14:03 |
| annegentle | I think ekarlso- has a question ahead of mine, but I need help troubleshooting [remote rejected] HEAD on a patch I'm trying to patch | 14:03 |
| dhellmann | Alex_Gaynor: looking... | 14:03 |
| *** frankdeez has quit IRC | 14:03 | |
| *** nayward has quit IRC | 14:04 | |
| dhellmann | Alex_Gaynor: that fix isn't specific to neutron client, although those developers are the ones that reported it | 14:05 |
| openstackgerrit | A change was merged to openstack/requirements: Update boto not use the version 2.13.0 https://review.openstack.org/46449 | 14:05 |
| dhellmann | Alex_Gaynor: there are no new upstream dependencies, it just fixes broken unicode handling under py26 | 14:05 |
| Alex_Gaynor | dhellmann: ok, sounds goood | 14:05 |
| fungi | ekarlso-: for corporate cla, you or your manager can probably start from https://wiki.openstack.org/wiki/HowToUpdateCorporateCLA | 14:05 |
| fungi | ekarlso-: or possibly the "corporations" bullet at https://wiki.openstack.org/wiki/How_To_Contribute#Contributors_License_Agreement | 14:05 |
| *** dcramer__ has joined #openstack-infra | 14:06 | |
| *** ArxCruz_ has joined #openstack-infra | 14:06 | |
| *** ArxCruz has quit IRC | 14:07 | |
| fungi | ekarlso-: you can also ask reed the next time he's on irc, or e-mail communitymngr@openstack.org with questions | 14:08 |
| *** dizquierdo has joined #openstack-infra | 14:08 | |
| *** dkliban has joined #openstack-infra | 14:08 | |
| fungi | ekarlso-: also, you should be aware that the corporate cla is mostly a formality. we don't have any mechanisms to prevent you from contributing as long as you've signed the individual cla | 14:10 |
| fungi | annegentle: can you paste.openstack.org the error you're getting along with a little more output for context? | 14:11 |
| *** dizquierdo has quit IRC | 14:11 | |
| annegentle | fungi: sure! http://paste.openstack.org/show/47005/ | 14:11 |
| ArxCruz_ | fungi: I dc, last message was <ArxCruz> actually, it's returning nothing, so I believe it's not getting the memorytotalbytes from modules/jenkins/lib/facter/memorytotalbytes.rb | 14:11 |
| annegentle | fungi: I've pushed from this repo before, it's on a cloud server | 14:11 |
| annegentle | fungi: and since I usually push from mac, not from ubuntu, I'm a little clueless | 14:12 |
| annegentle | fungi: but only a little | 14:12 |
| *** dizquierdo has joined #openstack-infra | 14:12 | |
| fungi | ArxCruz_: yeah, when i run 'facter memorytotalbytes' on random jenkins slaves of ours, i see i get a null string back | 14:12 |
| fungi | ArxCruz_: however that's also apparently facter's behavior when you pass it the name of a fact it doesn't know about | 14:12 |
| ArxCruz_ | fungi: so, I believe facter isn't getting memorytotalbytes | 14:12 |
| *** dcramer__ has quit IRC | 14:14 | |
| anteaya | annegentle: the part I am seeing is no changes made | 14:14 |
| annegentle | anteaya: ohhh so my commit isn't showing a diff | 14:14 |
| anteaya | my interpretation is that git review/gerrit believes there are no changes | 14:14 |
| anteaya | that's what I would say | 14:14 |
| annegentle | anteaya: got it | 14:15 |
| fungi | annegentle: yeah, rather gerrit thinks there are no changes between your current commit and the most recent patchset for that change | 14:15 |
| annegentle | anteaya: thanks | 14:15 |
| ekarlso- | fungi: ah ok ! | 14:16 |
| annegentle | so what I'm trying to do is make sure nova.conf.sample is updated, pip8 thinks I haven't run the script https://review.openstack.org/#/c/40271/ | 14:16 |
| anteaya | annegentle: np | 14:16 |
| *** niska has quit IRC | 14:16 | |
| *** dizquierdo has quit IRC | 14:17 | |
| *** dizquierdo has joined #openstack-infra | 14:17 | |
| *** dizquierdo has left #openstack-infra | 14:18 | |
| annegentle | I'm going to try from another machine | 14:18 |
| fungi | annegentle: what do git status and git show give you? | 14:18 |
| *** niska has joined #openstack-infra | 14:19 | |
| annegentle | fungi: yeah git status is blank | 14:20 |
| fungi | mainly want to make sure that git status doesn't indicate any unrepresented changes/untracked files in your current working copy, and that git show has something different than the latest patchset on that change | 14:20 |
| * anteaya dreams of being able to rent a dhellmann petting cat during layovers | 14:20 | |
| * dhellmann sees dollar signs | 14:20 | |
| anteaya | :D | 14:20 |
| annegentle | hee | 14:21 |
| mordred | ekarlso-: you just need to get your corp to sign one | 14:22 |
| mordred | ekarlso-: we don't have any enforcement around the corp agreements | 14:22 |
| *** lnxnut_ has joined #openstack-infra | 14:23 | |
| *** ruhe has quit IRC | 14:24 | |
| anteaya | I do believe ekarlso- works with hp | 14:26 |
| *** adalbas has quit IRC | 14:27 | |
| anteaya | so I think you are in good shape ekarlso- | 14:27 |
| *** thomasbiege has joined #openstack-infra | 14:29 | |
| *** thomasbiege has quit IRC | 14:30 | |
| * anteaya is afk for a bit | 14:31 | |
| *** yamahata has quit IRC | 14:31 | |
| *** ruhe has joined #openstack-infra | 14:32 | |
| *** rnirmal has joined #openstack-infra | 14:32 | |
| *** nayward has joined #openstack-infra | 14:34 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:35 | |
| Alex_Gaynor | the jenkins01 UI is pretty slow today/last night, anyone else seeing that? | 14:35 |
| Ajaeger1 | Alex_Gaynor: Yeah, noticed last night as well. | 14:36 |
| annegentle | fungi: so git show does show differences, but they're the same differences as the patchset itelf | 14:36 |
| annegentle | fungi: oh maybe I should rebase | 14:36 |
| Ajaeger1 | Alex_Gaynor: I really got to love the git os-log tool that was just announced ;) | 14:36 |
| *** acabrera has joined #openstack-infra | 14:36 | |
| annegentle | fungi: no that's not it. It's really that somehow pip8 tests don't think I ran the generate_sample.sh script | 14:37 |
| fungi | annegentle: it could be that generate_sample.sh is broken and not changing anything | 14:38 |
| * fungi tries | 14:38 | |
| mordred | dhellmann: did you announce os-log? | 14:40 |
| *** atiwari has joined #openstack-infra | 14:40 | |
| dhellmann | I posted it to planet.openstack.org, but haven't sent email to the dev list | 14:41 |
| *** nayward has quit IRC | 14:41 | |
| * dhellmann adds a todo to his list for the day | 14:42 | |
| *** adalbas has joined #openstack-infra | 14:45 | |
| *** markmcclain has joined #openstack-infra | 14:48 | |
| ArxCruz_ | fungi: figure out the problem | 14:48 |
| ArxCruz_ | puppet and facter installed on my agent is different from my puppet master :) | 14:49 |
| fungi | ArxCruz_: oh? i hadn't had time to play around with it any more | 14:49 |
| fungi | aha | 14:49 |
| fungi | good catch | 14:49 |
| ArxCruz_ | custom facts are only allowed on facter 1.7 | 14:49 |
| ArxCruz_ | i had facter 1.6 | 14:49 |
| ArxCruz_ | openstack infra use puppet from puppet labs, I've installed puppet from ubuntu | 14:49 |
| *** Ajaeger1 has quit IRC | 14:50 | |
| fungi | yeah, we use facter 1.7.1 on our precise systems | 14:50 |
| ArxCruz_ | fungi: documentation should be updated (if there isn't this information) | 14:52 |
| *** UtahDave has joined #openstack-infra | 14:52 | |
| ArxCruz_ | or in the manifests should ensure that puppet version is > 1.6 | 14:52 |
| fungi | ArxCruz_: yeah, i agree there are probably some undocumented assumptions that our modules expect puppet from puppetlabs (our bootstrapping scripts basically use the distro-provided puppet to upgrade to the puppetlabs one) | 14:53 |
| fungi | annegentle: when running tools/config/generate_sample.sh what options are you passing it? | 14:54 |
| annegentle | fungi: I do /tools/config/generate_sample.sh -b . -p nova -o etc/nova | 14:55 |
| annegentle | fungi: http://paste.openstack.org/show/47009/ | 14:55 |
| fungi | looks like it needs to run in a virtualenv with oslo.config installed, at a minimum | 14:55 |
| *** mrodden has quit IRC | 14:55 | |
| annegentle | fungi: ok I have a way to do that | 14:55 |
| annegentle | fungi: it's "python tools/install_venv.py" right? | 14:57 |
| fungi | yeah, i'm zeroing in on being able to test it | 14:57 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Map all savanna repos to savanna at launchpad https://review.openstack.org/37621 | 14:57 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/jeepyb: Improve git2lp and is_direct_release behaviour https://review.openstack.org/37620 | 14:57 |
| fungi | oh, maybe | 14:57 |
| fungi | i just manually created a virtualenv | 14:57 |
| annegentle | fungi: ok | 14:57 |
| annegentle | fungi: how? | 14:57 |
| fungi | and am pip installing stuff into it as i discover it's missing | 14:57 |
| annegentle | fungi: I also have run pip install -r nova/requirements.txt on this box | 14:58 |
| fungi | 'virtualenv samples ; . samples/bin/activate ; pip install oslo.config ...' | 14:58 |
| annegentle | fungi: ohhh | 14:58 |
| fungi | i have recently come to the realization that globally installing things on my dev virtual machines with pip is an invitation for pain | 14:58 |
| fungi | i just went ahead and did a pip install -r test-requirements.txt in the venv. i'll see if that's any better | 15:00 |
| fungi | nova really ought to have a tox target to run this script though. would make things much easier to be able to run tox -e samples | 15:00 |
| fungi | could then just pull in whatever's needed on the fly into a venv and pass the right options to the generate_samples.sh script | 15:01 |
| annegentle | fungi: yeah nova/requirements.txt hasn't worked for me yet | 15:01 |
| *** nayward has joined #openstack-infra | 15:02 | |
| annegentle | fungi: oh but in the vent. I'll try that. | 15:02 |
| fungi | i'm remembering that there are some debs i'll need to install on this vm to cover non-python deps like postgres and mysql and xml stuff | 15:02 |
| *** zeus has joined #openstack-infra | 15:02 | |
| *** zeus has quit IRC | 15:03 | |
| mordred | fungi: I am fully of the opinion that global installation of python code via pip in all situations is a recipe for pain | 15:04 |
| annegentle | fungi: oh! I think you got me unstuck anyway so thanks for shared pain! | 15:05 |
| fungi | annegentle: my pleasure | 15:05 |
| mordred | fungi: https://github.com/stankevich/puppet-python | 15:06 |
| annegentle | fungi: Maybe I'll meet the deadline for voting for the PTL of nova! LOL | 15:06 |
| fungi | annegentle: fingers crossed. i take it this is your first nova patch in a while? | 15:06 |
| annegentle | fungi: I usually get one in a release :) | 15:07 |
| mordred | fungi: I think maybe we shoudl think about using that in our puppet | 15:07 |
| fungi | mordred: reading through it, and at least on the surface it does sound like a rainbow-striped pony ice cream sundae | 15:07 |
| *** blamar has joined #openstack-infra | 15:08 | |
| annegentle | fungi: is there a correct way to get out of the venv? | 15:09 |
| fungi | annegentle: 'deactivate' | 15:09 |
| annegentle | fungi: while in the (.venv) blah blah prompt? | 15:09 |
| fungi | yep | 15:09 |
| annegentle | fungi: thanks | 15:09 |
| fungi | part of the virtualenv's activate shell stub adds a deactivate function into your environment | 15:10 |
| fungi | and running that tears down all the environment changes/additions made by the activate stub | 15:10 |
| * annegentle will slowly learn python slowly | 15:10 | |
| fungi | it is black magic, until you discover how badly you needed it, and then you use it until your fingers bleed | 15:11 |
| *** zeus has joined #openstack-infra | 15:11 | |
| *** zeus has quit IRC | 15:11 | |
| *** zeus has joined #openstack-infra | 15:11 | |
| annegentle | fungi: heh | 15:11 |
| *** nayward has quit IRC | 15:14 | |
| ttx | dhellmann: scratching a few itches, I see | 15:19 |
| *** chuckieb has quit IRC | 15:19 | |
| *** dcramer__ has joined #openstack-infra | 15:20 | |
| *** pblaho has quit IRC | 15:22 | |
| *** tstevenson_ has joined #openstack-infra | 15:32 | |
| openstackgerrit | A change was merged to openstack/requirements: Update the version of cliff in global-requirements.txt https://review.openstack.org/46439 | 15:33 |
| mordred | annegentle: yeah - I've been thinking for a while that all of our "help" around creating venvs magically is misplaced | 15:34 |
| mordred | annegentle: and that teaching people how they work would be more beneficial long term | 15:34 |
| UtahDave | fungi: Those changes looked great to me. What's the next step? Do we need to set up a jenkins job to execute that salt command? | 15:39 |
| *** mrodden has joined #openstack-infra | 15:40 | |
| fungi | UtahDave: i was waiting to see if i got more reviews, but looks like i have plenty so i'm merging them and then spinning up that slave in the next few minutes | 15:41 |
| UtahDave | great. Let me know how that goes and how I can help. | 15:41 |
| mordred | fungi: yeah - the code all looked great to me - I just wanted to make sure someone was going to be around to shepherd | 15:42 |
| fungi | mordred: that's the plan ;) | 15:42 |
| fungi | also making sure salt is upgraded on the master real quick since we don't use auto-upgrades on that machine for safety purposes | 15:43 |
| mordred | ++ | 15:44 |
| fungi | UtahDave: i notice that on our master we don't have salt-minion installed... is the master also a minion and, if so, do we need to get that added? | 15:46 |
| mordred | fungi: I think that we want our master to be a minion too, so that we can run commands there | 15:47 |
| UtahDave | A master is not also a minion. Many people do indeed run a minion on the master to manage the master as well | 15:47 |
| fungi | but as far as the tooling we're putting in place for the puppet update triggers, the master is able to run things locally as well as tell minions to run things? | 15:48 |
| fungi | (keeping in mind that what we're going to script is "update git repository on master" followed by "trigger puppet agent on slaves") | 15:49 |
| UtahDave | fungi: Ah, ok. Right now we're NOT set up to update the git repository on the master yet. Just trigger the puppet agent | 15:50 |
| mordred | fungi: I believe only minions can run things | 15:50 |
| fungi | so the master also need to be a minion to be able to tell itself to run something | 15:50 |
| fungi | s/need/needs/ | 15:51 |
| *** afazekas has quit IRC | 15:51 | |
| UtahDave | pretty much. The master does have the concept of a "runner" which is a command executed from the master itself, but I think this will work better if a minion is running on the master as well. | 15:51 |
| mordred | Alex_Gaynor: if I get this: No module named _bsddb | 15:52 |
| mordred | Alex_Gaynor: when trying to do tox -epypy | 15:52 |
| mordred | Alex_Gaynor: what am I doing wrong? | 15:52 |
| Alex_Gaynor | mordred: I haven't heard of this issue before, where'd you get your pypy from? | 15:52 |
| mordred | Alex_Gaynor: ubuntu | 15:52 |
| mordred | Alex_Gaynor: I was just trying to do tox -epypy on the pbr repo | 15:53 |
| Alex_Gaynor | mordred: meaning the PPA, or what's int he standard repos? | 15:53 |
| mordred | what's in the standard repos | 15:53 |
| Alex_Gaynor | That's like 1.7 or something reasonably old, I honestly don't remember what would cause that issue, I'd try with the PPA | 15:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Add saltstack APT repository to automatic upgrades https://review.openstack.org/46317 | 15:58 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Remove d2to1 usage https://review.openstack.org/46519 | 15:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Disable salt master and minions globally" https://review.openstack.org/46318 | 15:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a Jenkins slave for triggering salt commands https://review.openstack.org/46319 | 15:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Mention the --salt option in launch instructions https://review.openstack.org/46320 | 15:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Give Jenkins the ability to trigger puppet runs https://review.openstack.org/45898 | 15:59 |
| * anteaya is back and excited about Salt going in | 16:03 | |
| *** reed has joined #openstack-infra | 16:03 | |
| *** reed has quit IRC | 16:04 | |
| *** reed has joined #openstack-infra | 16:04 | |
| UtahDave | :) | 16:04 |
| *** senk has joined #openstack-infra | 16:05 | |
| *** yaguang has joined #openstack-infra | 16:05 | |
| fungi | in about another 10 minutes i'll start spinning up salt-trigger.slave.openstack.org and see what we get | 16:05 |
| *** pabelanger_ has joined #openstack-infra | 16:05 | |
| * anteaya rubs hands in anticipation | 16:06 | |
| Alex_Gaynor | fungi: is the intent to move from puppet to salt, or have both? | 16:06 |
| anteaya | both | 16:06 |
| anteaya | salt is here to tell puppet to run | 16:06 |
| anteaya | since right now puppet runs on a timer, not on events | 16:06 |
| anteaya | salt is here to listen for events and trigger puppet runs if certain events occur | 16:07 |
| *** zehicle has quit IRC | 16:07 | |
| anteaya | this set of patches triggers a general puppet run | 16:07 |
| anteaya | once we are happy with this set up, we will fine tune for specific puppet runs | 16:08 |
| anteaya | well strictly speaking Jenkins tells salt to tell puppet to run | 16:08 |
| *** matty_dubs is now known as matty_dubs|lunch | 16:09 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Ignore hacking warnings https://review.openstack.org/46481 | 16:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 16:09 |
| *** marun has quit IRC | 16:10 | |
| *** marun has joined #openstack-infra | 16:11 | |
| *** salv-orlando has left #openstack-infra | 16:11 | |
| Alex_Gaynor | anteaya: interesting | 16:11 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Move requirements install to a script https://review.openstack.org/46495 | 16:11 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Update projects when requirements change https://review.openstack.org/40068 | 16:11 |
| *** pabelanger_ has quit IRC | 16:12 | |
| *** pabelanger_ has joined #openstack-infra | 16:12 | |
| *** pabelanger has quit IRC | 16:12 | |
| *** pabelanger_ is now known as pabelager | 16:12 | |
| *** pabelanger has joined #openstack-infra | 16:12 | |
| mordred | Alex_Gaynor: I believe theoretically we'd be happier if all of our puppet was magically salt - problem is, we have a LOT of puppet that mostly works | 16:13 |
| *** pabelanger has quit IRC | 16:13 | |
| *** pabelager is now known as pabelanger | 16:13 | |
| mordred | Alex_Gaynor: so the value of re-writing it all is ... questionable | 16:13 |
| *** pabelanger_ has joined #openstack-infra | 16:13 | |
| *** mrodden has quit IRC | 16:16 | |
| *** mrodden1 has joined #openstack-infra | 16:16 | |
| *** pabelanger has quit IRC | 16:19 | |
| pleia2 | morning | 16:22 |
| mrodden1 | i tried using salt for config management for awhile, but went back to chef | 16:22 |
| mrodden1 | salt templating seems slow and injecting the config data wasn't really well laid out with the pillars system i thought | 16:23 |
| mrodden1 | its great for remote execution though | 16:23 |
| *** mrodden1 is now known as mrodden | 16:23 | |
| anteaya | morning pleia2 | 16:24 |
| *** ruhe has quit IRC | 16:25 | |
| *** SergeyLukjanov has quit IRC | 16:26 | |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Remove quantumclient from global requirements https://review.openstack.org/46533 | 16:26 |
| *** doctorjnupe has joined #openstack-infra | 16:28 | |
| *** Ajaeger has joined #openstack-infra | 16:28 | |
| *** jpich has quit IRC | 16:29 | |
| *** pabelanger has joined #openstack-infra | 16:30 | |
| *** reed has quit IRC | 16:31 | |
| *** afazekas has joined #openstack-infra | 16:31 | |
| mordred | dhellmann: if you merge this: https://github.com/dreamhost/sphinxcontrib-pecanwsme/pull/3 and release I will think very highly of you | 16:32 |
| mordred | dhellmann: (I'm scanning through anything in our requirements list which consumes d2to1) | 16:32 |
| *** comstud is now known as bearhands | 16:32 | |
| *** pabelanger has quit IRC | 16:32 | |
| dhellmann | mordred: ack | 16:33 |
| openstackgerrit | A change was merged to openstack-infra/reviewday: Update bin/reviewday to use neutron. https://review.openstack.org/46487 | 16:33 |
| clarkb | morning | 16:34 |
| mordred | dhellmann: also - https://review.openstack.org/46536 when you get a sec | 16:34 |
| mordred | clarkb: morning! | 16:35 |
| fungi | wow. been a while since anyone's manually run puppet agent on the master, looks like | 16:35 |
| clarkb | fungi: we turned it off when salt was being annoying | 16:35 |
| fungi | ahh | 16:35 |
| clarkb | fungi: I guess it didn't get turned back on when the change to make that not a problem was merged | 16:35 |
| anteaya | morning clarkb | 16:35 |
| clarkb | anteaya: good morning | 16:35 |
| *** zehicle has joined #openstack-infra | 16:36 | |
| clarkb | so in theory we have review.o.o backups now. I will confirm then we can summon the meteor | 16:36 |
| anteaya | yay for backups | 16:36 |
| mordred | woot | 16:36 |
| *** mkerrin has quit IRC | 16:39 | |
| pleia2 | dprince: I'm pretty familiar with reviewday these days, I'd be happy to do reviews+testing if you want to avoid your own review+approve every time | 16:45 |
| dprince | pleia2: cool. | 16:46 |
| *** erfanian has quit IRC | 16:46 | |
| *** acabrera is now known as acabrera_away | 16:46 | |
| * anteaya gets out her bristolboard for a "pleia2 for reviewday core" sign | 16:47 | |
| dprince | pleia2: I'd be happy to have you on the review team. Sorry I didn't nominate you sooner. | 16:48 |
| dprince | pleia2: openstack-infra admins have privs here as well but those guys are plenty busy so best jump on them ourselves. | 16:49 |
| dprince | where them == the reviews | 16:49 |
| pleia2 | dprince: *nods* that's why I figured I'd offer, I get email notifications for them but often Merged by the time I see them :) | 16:49 |
| dprince | pleia2: Okay. I've added you to group #123. openstack-reviewday | 16:52 |
| * dprince feels lucky to have that number | 16:52 | |
| pleia2 | hehe | 16:52 |
| * mordred starts calling dprince 123 | 16:53 | |
| *** CaptTofu has joined #openstack-infra | 16:53 | |
| * dprince changes all password suffixes | 16:53 | |
| *** yolanda has quit IRC | 16:55 | |
| *** reed has joined #openstack-infra | 16:56 | |
| * fungi grumbles... we need to pip install rackspace-novaclient (the the pyrax plugin to novaclient) on our puppet master | 16:56 | |
| *** matty_dubs|lunch is now known as matty_dubs | 16:57 | |
| *** tstevenson_ has quit IRC | 17:00 | |
| clarkb | fungi: whyfor? | 17:02 |
| fungi | clarkb: rackspace-specific authentication has been dropped in the new version of novaclient which the master just upgraded to when i ran the puppet agent | 17:02 |
| *** hemnafk is now known as hemna | 17:04 | |
| *** pcrews has quit IRC | 17:04 | |
| fungi | clarkb: actually, the alternative is to no longer set NOVA_RAX_AUTH and then replace the api key with the actual account password | 17:04 |
| mordred | yes | 17:05 |
| mordred | that is the thing that does not require rax auth | 17:05 |
| fungi | i'm not sure which we find least distasteful... pin to old novaclient, pip install pyrax extension (because there's no deb), or put our rackspace account management passwords on that server | 17:05 |
| *** yaguang has quit IRC | 17:05 | |
| *** acabrera_away is now known as acabrera | 17:05 | |
| fungi | opinions? | 17:06 |
| mordred | I see absolutely no issue with using passwords | 17:06 |
| mordred | it's how hpcloud already works | 17:06 |
| mordred | and in our usage, there's really no substantive difference between the rax api key and the rax password | 17:06 |
| fungi | we did theoretically gain some separation with the api key, because there were certain things you couldn't do via api, so it protected access to those webui account functions. but i agree that was probably more a side effect than intent | 17:07 |
| mordred | yah. also, we have separate accounts for separate purposes | 17:07 |
| fungi | however we need them all entered on the master, so the separate accounts argument is somewhat irrelevant to this topic (but would certainly be relevant in places like nodepool) | 17:08 |
| fungi | anyway, i agree changing to the actual rackspace passwords where we're currently using pyrax auth keys is probably fine, but i'm not going to do it without unanimous infra core consent | 17:10 |
| clarkb | wfm, ++ to seeing that jeblair has to say | 17:11 |
| clarkb | fungi: do we need to send a note ot hte mailing list about short gerrit downtime tomorrow? | 17:11 |
| fungi | yeah, maybe he'll find internets while in transit, though the passwords thing is not an urgent decision | 17:11 |
| fungi | clarkb: i suppose a little warning can't hurt, though i doubt anyone will notice since it will be very, very brief | 17:12 |
| fungi | i seem to remember not sending notices about these in the past unless we were moving several projects at once and/or break^Wchanging other stuff | 17:13 |
| *** nati_ueno has joined #openstack-infra | 17:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Update etherpad and etherpad puppet manifests. https://review.openstack.org/45360 | 17:15 |
| clarkb | fungi: ok | 17:15 |
| clarkb | 45360 will now redirect etherpad users to /p/padname | 17:15 |
| openstackgerrit | A change was merged to openstack/requirements: Update Ceilometer alembic requirement to 0.6.0 version https://review.openstack.org/46004 | 17:15 |
| clarkb | It makes export work on my test box https://etherpad-test.novalocal/p/clarkb-test with an IP of 15.185.109.39 if you want to test | 17:16 |
| zaro | anteaya: what's our plans for owncloud? | 17:17 |
| pleia2 | zaro: I believe a team had requested an instance of it for sharing files | 17:19 |
| anteaya | well my plans are to get it breathing | 17:19 |
| anteaya | I think the board wants it actually | 17:20 |
| anteaya | and wants it to be managed by infra | 17:20 |
| anteaya | there was some discussion yesterday about having the meeting calendar on owncloud | 17:20 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct Jenkins SSH key variable for salt-trigger https://review.openstack.org/46546 | 17:20 |
| * fungi sighs... ^ brown bag fix | 17:20 | |
| anteaya | so that ttx doesn't have to keep updating the google cal file by hand | 17:21 |
| anteaya | but so far, I can't see how owncloud allows mulitple people to edit the same file/calendar | 17:21 |
| clarkb | fungi: did I breka it? | 17:21 |
| fungi | clarkb: you broke nothing | 17:21 |
| anteaya | but that might be my fault in the configuration, I might not have it set up correctly yet | 17:21 |
| clarkb | oh good not the launch script | 17:21 |
| fungi | clarkb: just me being sloppy | 17:22 |
| anteaya | which is why I gave the infra team their own accounts so we can test drive it | 17:22 |
| clarkb | fungi: is that change correct? | 17:22 |
| fungi | clarkb: seems consistent with other manifests, for example mirror_slave.pp | 17:22 |
| clarkb | fungi: jenkins_ssh_public_key is in the class parameter list | 17:22 |
| clarkb | I don't see where ssh_key is coming from | 17:22 |
| fungi | oh, i got it backwards too. crap | 17:22 |
| clarkb | oh, yeah I see that in mirror_slave | 17:23 |
| anteaya | so right now I have a test instance up, I need to puppetize it and then we need to have it on the agenda of a meeting to discuss stuff like db backups and what server it should be on and stuff | 17:23 |
| zaro | anteaya: thx for the info. thought it might be used for file sharing. iirc, my last experience with webdav was that it didn't work with windows 7. but maybe board doesn't have to worry about that. | 17:24 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct Jenkins SSH key variable for salt-trigger https://review.openstack.org/46546 | 17:24 |
| *** oz_akan_ has joined #openstack-infra | 17:24 | |
| fungi | better ^ | 17:24 |
| anteaya | zaro: oh if you have a windows 7 set up to test it, I would be grateful for your feedback | 17:25 |
| anteaya | then if the decision is to go ahead with it we can document any windows 7 issues | 17:26 |
| *** dcramer__ has quit IRC | 17:26 | |
| *** vogxn has joined #openstack-infra | 17:27 | |
| fungi | clarkb: also https://review.openstack.org/44394 changed behavior because perms on /root/ci-launch were originally 0755... contemplating setting group to admin on that dir--opinion? | 17:28 |
| jog0 | clarkb: found some interesting elastic search behaviors | 17:29 |
| clarkb | jog0: interesting in a good way or interesting in ways that will make me very sad? | 17:29 |
| jog0 | both I guess | 17:30 |
| clarkb | I have a restored backup on review.o.o and have extracted one of the mysql dumps | 17:30 |
| clarkb | so \o/ I guess | 17:30 |
| clarkb | jog0: :/ | 17:30 |
| jog0 | * sometimes there are results in elastic search that don't match the log files themselves | 17:30 |
| *** CaptTofu has quit IRC | 17:30 | |
| clarkb | jog0: hmm that may be a bug in my logstash filters as they can modify the log entries | 17:30 |
| zaro | anteaya: i remembered that this was the workaround i used.. http://www.techiechips.com/web-folders-web-disk-webdav-problems-on-windows-7/ | 17:31 |
| clarkb | jog0: for example they don't provide the swift apachecombined logs verbatim, and mulitline events are squashed together (or we attempt to do that at least) | 17:31 |
| clarkb | jog0: submitting a bug with examples is probably a good idea, should look into that | 17:31 |
| zaro | anteaya: check this out.. http://www.techiechips.com/web-folders-web-disk-webdav-problems-on-windows-7/ | 17:32 |
| jog0 | clarkb: such as @message:"ConnectionFailed: Connection to neutron failed: Maximum attempts reached" AND @fields.filename:"console.html" AND @fields.build_change:"45424" | 17:32 |
| clarkb | jog0: do you have a link to the console log on logs.o.o handy? | 17:33 |
| zaro | anteaya: i don't have a win 7 computer myself, but my wife uses it at home. i can do a quick test for you if you that helps. | 17:33 |
| jog0 | that query produces a hit on patchset 8 works | 17:33 |
| jog0 | clarkb: http://logs.openstack.org/24/45424/8/check/gate-tempest-devstack-vm-neutron/5f4e9d1 | 17:33 |
| pleia2 | anteaya: my laptop still dual boots win7, so we can also have a look next week if you want | 17:33 |
| pleia2 | zaro: oh yeah, you're in seattle too! we can all have a look :) | 17:34 |
| anteaya | zaro pleia2 oh yes please, the chances someone on the board will be using windows is high | 17:34 |
| clarkb | jog0: but it isn't in the source log? | 17:34 |
| anteaya | zaro: thanks for those links, will read | 17:34 |
| jog0 | clarkb: the other behavior is the logs appear to get parsed line by line so its hard to figure out when its safe to do a query | 17:34 |
| jog0 | clarkb: I couldn't find it | 17:34 |
| * anteaya is printing boarding passes | 17:34 | |
| jog0 | and the test passed in that case | 17:34 |
| pleia2 | anteaya: yay :) | 17:34 |
| clarkb | jog0: yeah I can't find it either, this is interesting | 17:34 |
| pleia2 | I fly out sunday evening, won't be getting in until late though | 17:35 |
| * jog0 files a bug with infra | 17:35 | |
| anteaya | pleia2: email me once you are in town | 17:35 |
| clarkb | jog0: thank you | 17:35 |
| pleia2 | anteaya: will do | 17:35 |
| jog0 | I was thinking those may be from jobs that finished but somehow got restarted | 17:36 |
| anteaya | Saturday is raincoat shopping, rain all next week in Seattle | 17:36 |
| clarkb | jog0: the timestamp is way off too | 17:36 |
| anteaya | Sunday is getting tripleo up, or as best I can | 17:36 |
| * pleia2 adds raincoat to packing list | 17:36 | |
| clarkb | jog0: very possibly a bug in the log processing side | 17:36 |
| pleia2 | my laptop doesn't have enough ram for tripleo, so i'll be remoting in to my system at home, should be fun :) | 17:36 |
| anteaya | pleia2: yeah I am just doing it all on my new hpcloud account (thanks mordred) | 17:37 |
| pleia2 | anteaya: unfortunately hpcloud doesn't have nested kvm, so it's not easy (i've been working to do it with lxc) | 17:37 |
| anteaya | oh :( | 17:38 |
| anteaya | have you any notes? | 17:38 |
| jog0 | https://bugs.launchpad.net/openstack-ci/+bug/1225108 | 17:38 |
| uvirtbot | Launchpad bug 1225108 in openstack-ci "logstash results don't always match up with actual logs" [Undecided,New] | 17:38 |
| pleia2 | anteaya: it doesn't yet work with full tripleo, just the small section I'm testing (so far) | 17:38 |
| jog0 | clarkb: so the other issue I found is knowing when the logs are fully parsed for a job | 17:38 |
| pleia2 | anteaya: I haven't found a way to run full tripleo in an hpcloud instance yet | 17:39 |
| anteaya | pleia2: I'd look at anything you have | 17:39 |
| pleia2 | anteaya: my work for baremetal testing is coming together here: https://etherpad.openstack.org/tripleobaremetallxc2013 | 17:39 |
| anteaya | pleia2: okay you have saved my forehead from some brick imprints | 17:39 |
| pleia2 | but it's not exactly tripleo | 17:39 |
| jog0 | clarkb: besides that, I am making good progress, I have a few queries that are assocated with bugs and I can pick out of the gerrit stream when tempest failed | 17:39 |
| pleia2 | it draws heavily from triplo, but diverges for specific work I'm doing | 17:39 |
| anteaya | pleia2: there's some notes though and I'll read them, thanks | 17:39 |
| * anteaya nods | 17:39 | |
| clarkb | jog0: awesome, that is great news | 17:40 |
| clarkb | jog0: yeah knowning when logs will be fully parsed for a job is hard... the current system doesn't keep track of that anywhere | 17:41 |
| clarkb | jog0: it shouldn't be too bad to stream out when that is done though | 17:41 |
| jog0 | clarkb: that would be great | 17:41 |
| clarkb | trying to figure out where in the pipeline would be best, I think the gearman client on logstash.o.o should pay attention to work complete packets | 17:42 |
| clarkb | (it currently ignores them) | 17:42 |
| *** vogxn has quit IRC | 17:43 | |
| jog0 | neato | 17:45 |
| clarkb | jog0: http://logs.openstack.org/24/45424/8/check/gate-tempest-devstack-vm-neutron/0150828/console.html | 17:49 |
| clarkb | jog0: so the bug here is I am not keeping track of the job UUID in logstash so mapping back to the log archive is hard | 17:49 |
| *** vipul is now known as vipul-away | 17:49 | |
| clarkb | jog0: I am thinking I will just put the path to the logs in the logstash event directly | 17:49 |
| clarkb | I have updated the bug, that particular change should be simple. I had planned on upgrading logstash and elasticsearch after openstack on ales and as part of that need to rewrite some of the logstash gearman worker/client code. If I don't get to fixing this stuff before then I will roll this up into that | 17:52 |
| *** vipul-away is now known as vipul | 17:52 | |
| *** nicedice has joined #openstack-infra | 17:52 | |
| jog0 | clarkb: cool sounsd like that part isn't to hard | 17:54 |
| jog0 | as for the work compleate packets how do I consume that information? | 17:55 |
| mrodden | do i need to make a blueprint or bug for https://review.openstack.org/#/c/46417/ or is it fine how it is? | 17:55 |
| clarkb | jog0: that will require a larger change to the gearman client workers so that they keep each other in sync with what work is being done. Then the gearman client can send work complete info someplace else | 17:59 |
| *** sarob has joined #openstack-infra | 17:59 | |
| clarkb | mrodden: it is fine as is | 18:00 |
| jog0 | clarkb: now way I can poll ES until its ready? | 18:00 |
| mrodden | clarkb: ok cool, thanks | 18:00 |
| *** burt has quit IRC | 18:00 | |
| clarkb | jog0: you can poll ES | 18:00 |
| clarkb | jog0: I would do an expoential backoff to some upper bound if possible to avoid hammering ES | 18:01 |
| jog0 | right, what would I poll that would signify logs are parsed | 18:01 |
| clarkb | jog0: @tags:"console.html" AND ( @message:"Finished: SUCCESS" OR @message:"Finished: FAILURE" ) or similar | 18:03 |
| clarkb | jog0: Finished: SUCCESS and Finished FAILURE should be unique strings in the console logs and they are the last lines of those files | 18:03 |
| jog0 | I tried that didn't seem to corrilate with all lgos being parsed | 18:03 |
| clarkb | jog0: right it iwll only correlate to that particular file beingparsed | 18:04 |
| jog0 | didn't even do that :( | 18:04 |
| clarkb | really? | 18:04 |
| jog0 | yeah | 18:04 |
| clarkb | I wonder if elasticsearch will index out of order for throughput reasons | 18:04 |
| jog0 | this is what I am using: https://github.com/jogo/elasticRecheck/blob/master/elasticRecheck.py#L64 | 18:05 |
| jog0 | haven't confirmed it works well yet though | 18:05 |
| jog0 | as only good way to test this is with a live stream | 18:05 |
| jog0 | and development is slow these days | 18:05 |
| openstackgerrit | A change was merged to openstack-infra/config: Correct Jenkins SSH key variable for salt-trigger https://review.openstack.org/46546 | 18:06 |
| clarkb | jog0: were you using the query I pasted before? | 18:07 |
| clarkb | I have unfortunately made the whole system so asynchronous that no single point has an entire picture of what work has been done :/ | 18:07 |
| jog0 | clarkb: I used that query or something very similar | 18:10 |
| jog0 | let me retry that one | 18:10 |
| *** gyee has joined #openstack-infra | 18:12 | |
| clarkb | jog0: something that may work is @tags:"$FILE_YOU_CARE_ABOUT" AND NOT @tags:"_grokparsefailure" basically wait until you get some valid properly parsed data for that particular file | 18:12 |
| clarkb | jog0: that doesn't garuntee the entire file has been indexed, but it gets you much closer to the point where indexing happens | 18:13 |
| *** lnxnut_ has quit IRC | 18:13 | |
| jog0 | clarkb: yeah | 18:13 |
| clarkb | jog0: (also further restrict by change, patchset, and build) | 18:13 |
| jog0 | I sleep 60 may just be my solution | 18:13 |
| jog0 | I have the change and patchset constraint in | 18:13 |
| jog0 | see ^ | 18:13 |
| clarkb | jog0: maybe sleep(60) once @tags:"$FILE_YOU_CARE_ABOUT" AND NOT @tags:"_grokparsefailure" returns results | 18:14 |
| clarkb | and hopefully I can work out something better with gearman | 18:14 |
| *** senk has quit IRC | 18:15 | |
| *** dprince has quit IRC | 18:15 | |
| *** NobodyCam has quit IRC | 18:16 | |
| Alex_Gaynor | clarkb: not sure if you've noticed already, but jenkins01 seems pretty slow | 18:16 |
| clarkb | Alex_Gaynor: I hadn't noticed | 18:16 |
| clarkb | Alex_Gaynor: anything in particular is slow? job throughput, web ui? | 18:17 |
| clarkb | looks like web ui is slow | 18:17 |
| Alex_Gaynor | clarkb: web UI | 18:17 |
| *** bswrchrd has joined #openstack-infra | 18:17 | |
| Alex_Gaynor | not sure if there's any signifance, besides for people watching ongoing builds, but I thought I'd point it out | 18:17 |
| hub_cap | hey can someone point me to "that which configs and runs the tempest tests" ? im looking for some info on how heat builds their image for the orchestration tests | 18:17 |
| clarkb | hub_cap: https://git.openstack.org/cgit/openstack-infra/devstack-gate https://git.openstack.org/cgit/openstack-dev/devstack and https://git.openstack.org/cgit/openstack/tempest depending on what config is happening | 18:18 |
| clarkb | hub_cap: devstack-gate configures system level things then pushing settings into devstack and tempest | 18:18 |
| hub_cap | got it. ill peep d-g | 18:19 |
| *** pabelanger_ has quit IRC | 18:19 | |
| *** reed has quit IRC | 18:21 | |
| clarkb | Alex_Gaynor: I am digging into it, but nothing is immediately apparent. We are only 5MB into swap, cpu utilization isn't horrible and so on | 18:21 |
| clarkb | Alex_Gaynor: but it is definitely slow for me as well | 18:21 |
| Alex_Gaynor | clarkb: weird, does it effect the speed at which we run jobs? if not maybe we don't care? | 18:22 |
| clarkb | Alex_Gaynor: I don't think it has a major impact as most of the work is done on the slave. It will probably slow down some of the book keeping around jobs. I am still digging | 18:23 |
| *** melwitt has joined #openstack-infra | 18:29 | |
| *** zul has quit IRC | 18:32 | |
| clarkb | Alex_Gaynor: I think jenkins may not be cleaning up slave ssh connection threads cleanly so we end up with thousands of threads that are blocked. I am not sure how the jvm does thread scheduling but that may impact the ability of threads that are runnable to get cpu time | 18:34 |
| clarkb | fungi: https://jenkins01.openstack.org/monitoring?part=threads mind taking a look as well? | 18:35 |
| *** zul has joined #openstack-infra | 18:35 | |
| Alex_Gaynor | clarkb: they JVM doesn't schedule threads at all, they have a 1-1 mapping to OS threads | 18:35 |
| fungi | oh, that's rather fun | 18:36 |
| fungi | ha. waited for it to load and eventually got "The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression." | 18:37 |
| clarkb | we have 7.7k threads on jenkins01 and 490 on jenkins02 | 18:38 |
| clarkb | I am going to blame those threads for now | 18:38 |
| fungi | yeah, i see jenkins02 is nowhere near as large | 18:38 |
| fungi | restarted much more recently? | 18:38 |
| * fungi checks | 18:38 | |
| Alex_Gaynor | clarkb: sure, totally possible that either a) the OS isn't scheduling them well for some reason, b) somehow the blocked threads are contending for an important lock in a way that makes it slower for active threads to use it | 18:39 |
| openstackgerrit | A change was merged to openstack/requirements: Remove quantumclient from global requirements https://review.openstack.org/46533 | 18:39 |
| fungi | started about 3 hours apart, several weeks ago | 18:40 |
| fungi | and we're theoretically balancing jobs/slaves between them | 18:40 |
| fungi | once i get the webui for 01 to load for me anyway... got as far as openid this time at least | 18:42 |
| fungi | Threads on jenkins01.openstack.org@166.78.188.99: Number = 7,800, Maximum = 7,818, Total started = 796,297 | 18:43 |
| fungi | Threads on jenkins02.openstack.org@166.78.48.99: Number = 480, Maximum = 1,289, Total started = 756,190 | 18:43 |
| fungi | Threads on jenkins.openstack.org@198.101.231.203: Number = 74, Maximum = 130, Total started = 19,196 | 18:43 |
| clarkb | `jstack -l -J-d64 $JENKINS_PID` then shutdown mode and restart the service? | 18:43 |
| clarkb | that seem sane? | 18:43 |
| fungi | begs the question... if we've started roughly the same number of threads on both and they've been running for roughly the same length of time, why does one have a couple orders of magnitude more running threads? | 18:44 |
| fungi | the max 02 ever seems to have gotten to is still about 20% what 01 is at now | 18:45 |
| clarkb | fungi: because they are getting stuck for some reason. Alex_Gaynor's lock thing could be the issue | 18:45 |
| clarkb | fungi: if one thread derped holding a lock then everyone loses | 18:45 |
| clarkb | apparently jstack is in the jdk and not jre packages... | 18:45 |
| fungi | got it. so random happenstance basically. could have been either (or both) of them, just 01 seems to have gotten "lucky" at some point recently and stopped being able to gc them | 18:46 |
| clarkb | apparently kill -3 PID may work | 18:46 |
| clarkb | I am going to test on jenkins-dev | 18:46 |
| fungi | so, yeah, stack dump and then graceful shutdown 01 and then kill it if needed before starting up clean | 18:46 |
| fungi | also worth remembering, it was discovered in our last extended jenkins thread debugging session that when it eventually dies it will also log stack traces of the threads it couldn't stop cleanly | 18:47 |
| clarkb | kill -3 seems to have worked. it dumped things into the jenkins.og | 18:48 |
| fungi | so we may find plenty of detail there without even needing to crack open the full fdump | 18:48 |
| clarkb | fungi: I am going to put jenkins01 into shutdown mode now, when it is done running jobs, I will hit it with kill -3 then restart | 18:49 |
| fungi | k | 18:49 |
| *** dcramer__ has joined #openstack-infra | 18:50 | |
| clarkb | mordred: jeblair: anything else you would like to have done before we kick it? | 18:51 |
| fungi | i see it in shutdown mode now. cool | 18:51 |
| mordred | clarkb: a rain dance? | 18:52 |
| fungi | goat sacrifice | 18:53 |
| clarkb | food is here apparently, /me eats while waiting for shutdown to happen | 18:55 |
| *** atiwari has quit IRC | 18:58 | |
| *** pabelanger has joined #openstack-infra | 18:59 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Make ci-launch directory readable by admin users https://review.openstack.org/46557 | 19:03 |
| *** dcramer__ has quit IRC | 19:03 | |
| fungi | i'm going to be popping out to grab an early dinner in about an hour, then back on to wrap up more work | 19:03 |
| clarkb | ok | 19:05 |
| clarkb | we can kick it in ~15 minutes if that works for you | 19:05 |
| clarkb | fungi ^ | 19:05 |
| fungi | yup | 19:06 |
| fungi | there was one job left when i looked a moment ago, but the webui is obviously teh flakiez | 19:06 |
| fungi | many, many proxy timeouts trying to load it | 19:07 |
| annegentle | any coremudgeons want to talk about content shared between repos? | 19:08 |
| annegentle | we've got to figure it out | 19:10 |
| fungi | annegentle: sure. in a perfect world, there should be no content shared between repos. in the non-perfect world in which we currently operate, things like oslo-incubator exist to make doing that slightly less painful | 19:10 |
| annegentle | fungi: oh ok | 19:10 |
| annegentle | fungi: that's a library tho | 19:10 |
| fungi | nope | 19:10 |
| annegentle | fungi: one idea we had would be to modify the maven plugin so that it outputs an artifact that lives on forever that doc builds can always get to | 19:11 |
| fungi | oslo-incubator is content (code yes, but still content) copied between projects using some scripts to keep them mostly in sync | 19:11 |
| annegentle | fungi: oh. what's the "mostly" | 19:11 |
| fungi | the "mostly" is they're only in sync when people manually pull new versions of the openstack common code into their projects | 19:12 |
| fungi | but at least there are defined and documented ways to do it consistently | 19:12 |
| annegentle | ok yeah I envision writers would manually pull in new versions of the artifact | 19:12 |
| fungi | if that content is only needed at build time, one way would be for the build process to pull a copy of the other project containing the content and then include that, so that it's not carted around in both projects and doesn't need to be updated in two places | 19:13 |
| pleia2 | sarob: having a look at your issue (but I will need other -infra help here) - what is your remote origin? (I'm assuming it would be github, but you're saying it's not updating for you even though github is correct) | 19:14 |
| *** nicedice has quit IRC | 19:14 | |
| *** nicedice has joined #openstack-infra | 19:14 | |
| pleia2 | sarob: git not updating for you is quite odd, the online documentation not updating seems like an issue with the post-processing of those docs | 19:14 |
| fungi | annegentle: and then both projects could share that build job, so if either one of them changes, the content gets built | 19:14 |
| annegentle | fungi: oh! lightbulb | 19:15 |
| pleia2 | (peanut gallery: see -infra list) | 19:15 |
| annegentle | fungi: but, that's only at the project level, not a specific directory? | 19:15 |
| jog0 | clarkb fungi: so I just noticed https://review.openstack.org/#/c/46169/ merged but it appears recheckwath didn't restart | 19:15 |
| clarkb | annegentle do remote includes not work? | 19:16 |
| fungi | annegentle: well, we can have jobs defined to only trigger on changes to specific file path patterns | 19:16 |
| *** alexpilotti has quit IRC | 19:16 | |
| annegentle | clarkb: sure they do, do you mean like http://raw.github.com/blah | 19:16 |
| fungi | clarkb: i think part of the challenge with remote includes is just in making sure your content gets rebuilt when the remote-included stuff changes as well | 19:16 |
| annegentle | clarkb: but we got burned by that when keystone's repo changed, and I imagine it'll be the same thing for incubated > integrated etc | 19:17 |
| clarkb | fungi annegentle gotcha | 19:17 |
| annegentle | fungi: yeah that's pretty much the crux | 19:17 |
| jog0 | but it appears recheck page is working otherwise | 19:18 |
| fungi | annegentle: so if it's one openstack project including content from another openstack project, we have ways to access those during jobs without getting burned by github being unreachable and what have you | 19:18 |
| annegentle | fungi: ok! | 19:19 |
| fungi | annegentle: including ideas currently being floated to cache repositories on the build slaves, to reduce the time needed to clone them from our infra | 19:19 |
| annegentle | fungi: I mean, I'm okay with name changes meaning those scripts get updated etc | 19:19 |
| fungi | jog0: having a look | 19:19 |
| annegentle | if Jenkins can't find nose_results.html, is it ok to do 'recheck no bug' on a nova patch? | 19:20 |
| clarkb | annegentle that wont cause a failure | 19:22 |
| clarkb | is a job failing? can you link to it? | 19:22 |
| annegentle | clarkb: yeah http://logs.openstack.org/71/40271/7/check/gate-nova-python27/c554da6/console.html | 19:22 |
| annegentle | clarkb: fungi helped me with venv earlier and I got pep8 passing, but then the python 2.7 and 2.6 failed | 19:23 |
| fungi | clarkb: looks like the last job on 01 finished... safe to move forward there? | 19:27 |
| *** gyee has quit IRC | 19:28 | |
| *** jcoufal has quit IRC | 19:29 | |
| clarkb | fungi ya give me a minute. gonna get back to laptop | 19:29 |
| clarkb | fungi: ready? | 19:32 |
| fungi | yup | 19:32 |
| clarkb | dumping jenkins01 thread stacks now | 19:32 |
| fungi | jog0: it didn't restart because we don't subscribe the recheckwatch service to changes on those files. change coming up | 19:32 |
| clarkb | fungi: we don't subscribe but we notify | 19:33 |
| clarkb | fungi: I think it should've restarted, unless this is a different file | 19:34 |
| clarkb | fungi: kill -3 was successful, I am going to restart jenkins service now | 19:34 |
| fungi | oh, i didn't think to look for notifies in there | 19:34 |
| fungi | okay, sounds good | 19:34 |
| clarkb | jenkins doesn't want to die, hitting it with kill -9 | 19:35 |
| fungi | clarkb: jog0: in fact, we do notify the service on changes to both of those files | 19:35 |
| fungi | clarkb: figures | 19:35 |
| fungi | clarkb: might be worth trying a hup and a segv after the term before going for full-on kill, but probably would ignore them under the current circumstances anyway | 19:36 |
| clarkb | fungi: it is starting back up now | 19:37 |
| fungi | k | 19:37 |
| *** dmakogon_ has joined #openstack-infra | 19:38 | |
| fungi | is puppet smart enough to restart a service on notify when there's no reload target in the initscript? | 19:38 |
| clarkb | now I need to recheck something | 19:38 |
| clarkb | fungi: yes I think it will use stop start | 19:38 |
| * fungi digs in the logs | 19:38 | |
| clarkb | fungi: but it may not, you can check the puppet type reference | 19:38 |
| clarkb | annegentle: Can you recheck no bug your nova change | 19:38 |
| annegentle | sure | 19:39 |
| clarkb | annegentle: I haven't had a chance to really debug those logs but I need to run jobs on jenkins01 | 19:39 |
| clarkb | jobs have started on jenkins01 | 19:39 |
| fungi | mmm... Sep 12 19:00:48 zuul puppet-agent[12549]: (/Stage[main]/Recheckwatch/Service[recheckwatch]) Triggered 'refresh' from 1 events | 19:39 |
| fungi | oh, actually the relevant one was... Sep 13 05:33:52 zuul puppet-agent[12549]: (/Stage[main]/Recheckwatch/Service[recheckwatch]) Triggered 'refresh' from 2 events | 19:40 |
| fungi | no errors from puppet when it did that | 19:41 |
| clarkb | Alex_Gaynor: can you do whatever you were doing on jenkins01 that was slow and see if it is better? | 19:42 |
| Alex_Gaynor | clarkb: yup, seems faster | 19:42 |
| sarob | pleia2: not sure if other probs are contrib | 19:46 |
| sarob | but the result is bad | 19:46 |
| *** lnxnut has joined #openstack-infra | 19:46 | |
| fungi | ahh, right, we have set hasrestart to true and haven't set hasstatus which defaults to true... and running that initscript with a status parameter returns "recheckwatch is not running" even though it's present in the process list | 19:46 |
| fungi | so i think puppet won't try to restart it until we fix the initscript | 19:47 |
| *** thomasm has quit IRC | 19:47 | |
| sarob | annegentle: any ideas why https://review.openstack.org/#/c/46182/ would be merged, in github, but not avail through git pull? | 19:47 |
| sarob | maybe i hit my head and im missing something simple | 19:48 |
| * sarob dizzy feeling that merges happen but dont | 19:49 | |
| Ajaeger | sarob, it's available through git pull . | 19:49 |
| pleia2 | sarob: I was asking about your origin because I'm able to clone it fine | 19:49 |
| Ajaeger | What are you missing? | 19:50 |
| pleia2 | both from git.openstack.org and github | 19:50 |
| Ajaeger | Are you looking at the right place? You added back again in that patch doc/src/docbkx | 19:50 |
| annegentle | sarob: oh yeah I've been wanting to talk to you about that | 19:52 |
| annegentle | clarkb: fungi ^^ we can talk to sarob about subprojects | 19:52 |
| annegentle | we can/ can we | 19:52 |
| *** jcoufal has joined #openstack-infra | 19:52 | |
| *** dripton has quit IRC | 19:53 | |
| fungi | subprojects e.g. git submodules | 19:53 |
| fungi | we don't (currently can't) support them | 19:53 |
| fungi | so whatever you read suggesting use of git submodules was leading you down the wrong path | 19:54 |
| annegentle | sarob: what did you need to do? | 19:55 |
| sarob | so i git pull origin master and get the old code from https://github.com/openstack/openstack-manuals | 19:55 |
| annegentle | sarob: for one, doc/src/docbkx/ went away Sunday | 19:56 |
| sarob | sure | 19:56 |
| annegentle | sarob: so that part's wrong | 19:56 |
| *** dripton has joined #openstack-infra | 19:57 | |
| annegentle | sarob: and are you looking for the other pages? | 19:57 |
| sarob | the patch has the correct dir | 19:57 |
| sarob | and https://github.com/openstack/openstack-manuals has the correct code and dir | 19:57 |
| sarob | my local repo has the wrong code and correct dir | 19:58 |
| Ajaeger | sarob, https://review.openstack.org/#/c/46182/ shows the wrong dir | 19:58 |
| Ajaeger | "Adoc/src/docbkx/openstack-training/sources/api-site" | 19:58 |
| *** rnirmal has quit IRC | 19:59 | |
| sarob | thats two left over junk source dir that are empty | 19:59 |
| sarob | i meant to delete them, but missed it | 19:59 |
| sarob | is missing the correct HTML http://docs.openstack.org/trunk/openstack-training/content/ | 20:00 |
| Ajaeger | sarob, they are not empty - they contain some submodules. | 20:00 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Make node SSH timeout configurable. https://review.openstack.org/46565 | 20:00 |
| Ajaeger | sarob, if they are wrong, could you +2 the following, please: https://review.openstack.org/#/c/46342/ | 20:00 |
| clarkb | jeblair: ^ curious to see what you think about 46565 | 20:01 |
| fungi | clarkb: jog0: seems the pidfile was nonexistent for some strange reason, which is why puppet managed not to restart it i think. manually killing it and then starting it created the correct pidfile, and restarting also seems to replace the pidfile as expected, so no idea why it was awol | 20:01 |
| clarkb | fungi: fun | 20:01 |
| clarkb | fungi: thank you for checking | 20:01 |
| sarob | ajaeger: done | 20:01 |
| *** tstevenson has quit IRC | 20:01 | |
| clarkb | annegentle: yes can talk about sarob's sub projects I am catching up on scrollback after hacking 46565 | 20:02 |
| fungi | clarkb: jog0: in theory it will restart on its own in the future, but i'll keep an eye out for a recurrence of this | 20:02 |
| clarkb | annegentle: where is a good place to start? | 20:03 |
| annegentle | sarob: clarkb so I think it was just a mistake (sean trying to use subprojects) | 20:03 |
| clarkb | annegentle: subproject == submodule in this case? | 20:04 |
| annegentle | clarkb: yes | 20:04 |
| sarob | ajaeger: it was left over from the infra team requesting that we not include from raw.github.com | 20:04 |
| annegentle | sarob: can you explain to -infra what you want to do (pull in chunks of content from other OpenStack repos) | 20:04 |
| annegentle | clarkb: sarob's use case is very similar to the API docs need | 20:05 |
| clarkb | sarob: is this related to needing an alternative to including from raw.github.com? | 20:05 |
| sarob | annegentle: correct | 20:05 |
| sarob | clarkb: correct | 20:05 |
| clarkb | sarob: can you just use git.openstack.org ? (this has the problems that annegentle and fungi described earlier but it is better than github) | 20:06 |
| clarkb | http://git.openstack.org/cgit/openstack/openstack-manuals/plain/README.rst for example | 20:06 |
| annegentle | clarkb: how do we point directly to a file? I couldn't figure it out | 20:06 |
| annegentle | clarkb: is plain = raw? | 20:06 |
| clarkb | annegentle: yes | 20:06 |
| *** atiwari has joined #openstack-infra | 20:06 | |
| annegentle | sarob: that should work | 20:06 |
| annegentle | sarob: as long as you're keeping up with the master branch | 20:06 |
| sarob | clarkb: yes, we would prefer git.openstack.org for now, but we would need raw version include xml | 20:07 |
| annegentle | clarkb: can you send an example of a way to point to a stable/grizzly file? | 20:07 |
| clarkb | http://git.openstack.org/cgit/openstack/openstack-manuals/plain/README.rst?h=stable/grizzly | 20:07 |
| sarob | got it | 20:07 |
| sarob | ill fiddle with it | 20:08 |
| fungi | anteaya: UtahDave: clarkb: salt-trigger.slave.openstack.org is up and puppeted successfully. i'm popping out for dinner and then will start testing it when i get back to an internet | 20:08 |
| clarkb | fungi: awesome | 20:08 |
| annegentle | clarkb: sweet | 20:08 |
| anteaya | fungi: yay | 20:08 |
| UtahDave | fungi: great to hear! | 20:09 |
| fungi | also, thanks for your help, UtahDave | 20:10 |
| clarkb | annegentle: sarob: fwiw you can use git urls like that to get similar functionality to what submodules provide | 20:10 |
| UtahDave | my pleasure, fungi. Keep me in the loop so I can help make sure everything runs smoothly | 20:10 |
| clarkb | annegentle: sarob: you can specify specific revisions. submodules won't solve the other project updated we need to rebuild this one and it won't solve the rename problem | 20:10 |
| clarkb | annegentle: sarob: Instead I think you may need to make use of zuul to trigger jobs across projects when there are dependencies | 20:11 |
| annegentle | clarkb: yeah I figured on both accounts | 20:11 |
| annegentle | clarkb: what does "make use of zuul" actually look like? | 20:11 |
| clarkb | eg repo1 depends on repo2 when repo2 merges a change put repo1's job in the post queue for repo2 | 20:11 |
| annegentle | clarkb: in the jenkins job yaml? | 20:12 |
| *** acabrera has left #openstack-infra | 20:12 | |
| clarkb | annegentle: in the layout.yaml | 20:12 |
| annegentle | clarkb: okie | 20:13 |
| clarkb | annegentle: http://paste.openstack.org/show/47041/ | 20:13 |
| annegentle | clarkb: this is actually looking do-able. | 20:13 |
| clarkb | notice I put the ops guide jobs in the manuals' post queue | 20:13 |
| *** nicedice has joined #openstack-infra | 20:13 | |
| annegentle | clarkb: I'll see if Laura Aves is up for it | 20:13 |
| sarob | Ajaeger pleia2: the pom dir change to the new dir structure got missed | 20:13 |
| annegentle | Aves/Alves | 20:13 |
| sarob | thanks guys | 20:14 |
| *** MarkAtwood has joined #openstack-infra | 20:14 | |
| MarkAtwood | hi. help with infra graphite? | 20:14 |
| annegentle | sarob: thank you! Hope your sprint day went well. | 20:14 |
| Ajaeger | sarob: what exactly got missed? | 20:14 |
| clarkb | MarkAtwood: sure, what's up? | 20:14 |
| annegentle | sarob: we've been doing a lot of follow up patching on those pom.xml files | 20:15 |
| MarkAtwood | the following url shows landed patches across all projects, per day http://graphite.openstack.org/render/?from=-2400hours&until=now&height=600&width=1000&lineMode=staircase&areaMode=all&hideLegend=yes&colorList=0096d6&target=summarize(stats_counts.gerrit.event.change-merged,'24h') | 20:15 |
| Ajaeger | sarob: the guide gets published at http://docs.openstack.org/trunk/training-guide/content/ | 20:15 |
| Ajaeger | I wouldn't be surprised if something is wrong - just want to know what ;) | 20:15 |
| MarkAtwood | i would like to generate similar graph, but for just one project, such as nova, or for trove | 20:15 |
| Ajaeger | canonical-url is indeed wrong | 20:16 |
| * Ajaeger fixes... | 20:16 | |
| Ajaeger | sarob: anything else? | 20:16 |
| clarkb | MarkAtwood: I am not sure our graphite is keeping track of that | 20:16 |
| clarkb | MarkAtwood: let me dig into the dirs of numbers | 20:16 |
| MarkAtwood | thanks clarkb | 20:17 |
| sarob | ajaeger annegentle: nope, im good. during the sprint day, i changed the pom, but with all the back and forth, i never merged the change | 20:17 |
| *** odyssey4me has quit IRC | 20:17 | |
| * sarob sigh | 20:17 | |
| MarkAtwood | there is probably just a way to compute a time series just the git history, but i dont have a tool to do it handy | 20:17 |
| clarkb | MarkAtwood: there is, git makes it a little tricky | 20:18 |
| clarkb | MarkAtwood: I am not seeing it in graphite, we seem to only deal with the aggregate there. Let me see what I can do with git | 20:18 |
| Ajaeger | sarob, so you take care of it? Great! | 20:18 |
| sarob | ajaeger: in the process... cheers | 20:19 |
| MarkAtwood | btw, what the heck happened the first half of august? we broke 400 patches per day on the monday of the the second week of august | 20:19 |
| MarkAtwood | clarkb, do you have home grown scripts for pulling it from git, or are their public git analysis tools? i didnt see anything that could do a time series from the git-dm project | 20:21 |
| clarkb | MarkAtwood: I don't have any tools. There is a visualization tool that walks the graph showing you stuff over time and I think that may also spit out raw numbers | 20:21 |
| annegentle | sarob: sorry I wasn't around Wed. to review your patches, that's one way where a second pair of eyes really helps | 20:22 |
| annegentle | way/place | 20:22 |
| annegentle | I can't type today | 20:22 |
| *** tstevenson has joined #openstack-infra | 20:24 | |
| MarkAtwood | thanks clarkb , what is that visualization tool? | 20:26 |
| *** Ajaeger has quit IRC | 20:28 | |
| clarkb | MarkAtwood: gource | 20:28 |
| *** dstufft has quit IRC | 20:28 | |
| *** dkliban has quit IRC | 20:29 | |
| *** rnirmal has joined #openstack-infra | 20:30 | |
| *** sarob has quit IRC | 20:34 | |
| *** sarob has joined #openstack-infra | 20:34 | |
| *** sarob has quit IRC | 20:37 | |
| *** sarob has joined #openstack-infra | 20:39 | |
| *** dstufft has joined #openstack-infra | 20:43 | |
| *** tstevenson_ has joined #openstack-infra | 20:48 | |
| *** tstevenson has quit IRC | 20:52 | |
| *** wenlock has joined #openstack-infra | 20:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 20:53 | |
| jog0 | fungi: thanks it looks good now | 20:53 |
| *** marun has quit IRC | 20:56 | |
| *** rfolco has quit IRC | 20:58 | |
| *** w_ has joined #openstack-infra | 20:59 | |
| *** olaph has quit IRC | 21:00 | |
| *** ruhe has joined #openstack-infra | 21:02 | |
| wenlock | hi, question about setting up jeepy , where does the gerrit-user get created in the modules under the ci project? can't seem to find that... is that manual? | 21:02 |
| *** matty_dubs is now known as matty_dubs|gone | 21:02 | |
| *** KLevenstein has quit IRC | 21:04 | |
| *** senk has joined #openstack-infra | 21:07 | |
| clarkb | wenlock: the user that jeepyb uses to connect to gerrit? | 21:08 |
| clarkb | wenlock: that user is manually created | 21:08 |
| *** beagles has quit IRC | 21:08 | |
| wenlock | ahh ok, i was banging my head on my desk trying to figure out why it didn't exist | 21:08 |
| wenlock | mt | 21:08 |
| *** beagles has joined #openstack-infra | 21:14 | |
| *** UtahDave has quit IRC | 21:14 | |
| *** openstackgerrit has quit IRC | 21:14 | |
| *** thedodd has quit IRC | 21:15 | |
| *** UtahDave has joined #openstack-infra | 21:15 | |
| jog0 | clarkb: running the first full pass at the recheck elstic search code | 21:18 |
| jog0 | should be interesting | 21:18 |
| clarkb | jog0: fingers are crossed | 21:19 |
| jog0 | clarkb: hard part is keeping up with the latest failing bug | 21:20 |
| jog0 | I only have two queries and not sure if those bugs have been seen in a while | 21:20 |
| *** mriedem has quit IRC | 21:20 | |
| *** ruhe has quit IRC | 21:21 | |
| clarkb | MarkAtwood: http://activity.openstack.org/dash/browser/repository.html?repository=nova.git gives you weekly numbers | 21:23 |
| clarkb | not sure what those numbers actually represent (eg how they are generated) | 21:24 |
| clarkb | jog0: I see something making queries | 21:25 |
| *** thedodd has joined #openstack-infra | 21:27 | |
| *** flaper87 is now known as flaper87|afk | 21:27 | |
| *** sarob_ has joined #openstack-infra | 21:28 | |
| *** tstevenson_ has quit IRC | 21:29 | |
| *** sarob has quit IRC | 21:31 | |
| jog0 | clarkb: heh thats me by hand | 21:32 |
| jog0 | nothing is failing gate right now :( | 21:32 |
| *** sarob_ has quit IRC | 21:32 | |
| jog0 | at least not tempest | 21:32 |
| *** ArxCruz_ has quit IRC | 21:33 | |
| jog0 | heh zuul has only 7 patches in flight right now | 21:36 |
| *** oz_akan_ has quit IRC | 21:37 | |
| clarkb | fungi: signing pypi packages. Any thoughts on how we can do that sanely? We will need to sign the packages that we want to push to maven nexus | 21:38 |
| *** thedodd has quit IRC | 21:39 | |
| *** rnirmal has quit IRC | 21:40 | |
| fungi | clarkb: so... there's a dilemma there | 21:42 |
| fungi | clarkb: we can sign things with a machine once they're created before they get sent somewhere, but then it's just the key on some machine attesting to something | 21:42 |
| *** tstevenson_ has joined #openstack-infra | 21:42 | |
| Alex_Gaynor | fungi: if the pypi package is built from a signed tag, that's at least better than nothing | 21:43 |
| Alex_Gaynor | dstufft might have some opinions/ideas | 21:43 |
| fungi | alternatively, we can generate things, wait for a human to inspect and sign them, then take that signature submitted somehow (probably code review to a special repository for that purpose) and upload once it's merged and validated | 21:43 |
| Alex_Gaynor | code review of a signed commit, obviously :P | 21:44 |
| clarkb | fungi: yeah that is what bothers me | 21:44 |
| dstufft | oh man trust is the hardest thing | 21:44 |
| clarkb | fungi: imo value of jenkins slave signing $thing is low | 21:44 |
| dstufft | literally the hardest | 21:44 |
| fungi | Alex_Gaynor: yes, i already have a routine worked out where we can pull a signed tag, verify it, pull the corresponding published tarball, compare it to the repo state which the signed tag represents looking for any differences, and then sign the tarball | 21:44 |
| clarkb | fungi: we could just do it to make maven happy >_> | 21:44 |
| fungi | but really, when a machine signs something, that means it at least has the (in openpgp terms) signing subkey available to it... so if that machine is subverted the key can theoretically be leveraged to sign things as that machine. same story for human-managed keys of course, but, there's people we can blame i guess? ;) | 21:46 |
| fungi | so, taking linux distros as an example, automated signing by machines with access to specific keys is commonplace | 21:47 |
| fungi | we can decide that's sane, and do it. sure. just needs some discussion | 21:48 |
| clarkb | zaro: ^ | 21:49 |
| *** prad has quit IRC | 21:49 | |
| fungi | for the tarballs i've been contemplating a separate tarball-attestation repository where people can submit their detached signatures of tarballs to which they're attesting. we could put those through code review and everything if we wanted | 21:49 |
| clarkb | I think I am going to hold off on setting up sonatype accounts then | 21:49 |
| clarkb | fungi: 1700UTC is 10am PDT right? | 21:50 |
| *** weshay has quit IRC | 21:51 | |
| fungi | we could trigger upload jobs off that repo if we wanted, so the workflow would be (1) sign and push tag, (2) wait for tarball to appear, (3) retrieve/check/sign tarball and submit detached signature, (4) review and merge signature, (5) upload tarball+signature | 21:51 |
| fungi | 10am pdt, yes | 21:51 |
| fungi | 1pm edt | 21:52 |
| fungi | on the other hand if we go the hands-off approach, the upload slave could retrieve the tarball and the git tag to which it corresponds, check the git tag against an authorized keychain and compare the unpacked tarball to the repository state, then sign the original packed tarball if it looks sane, then upload the tarball and machine-generated signature together | 21:55 |
| fungi | i like the latter approach since there's more automation and less human-introduced delay, but it also means convincing people that the upload slave and its signing key are safeguarded sufficiently to be trustworthy | 21:57 |
| fungi | and yes, i will make sure to tack Alex_Gaynor and dstufft onto any forthcoming reviews implementing these sorts of signing automation. more eyes is more better | 21:58 |
| clarkb | ++ | 22:00 |
| *** SergeyLukjanov has quit IRC | 22:00 | |
| clarkb | pleia2: lifeless: would now be a good time to start the tripleo testing discussion on that etherpad? https://etherpad.openstack.org/tripleo-initial-testing | 22:02 |
| *** bswrchrd has quit IRC | 22:04 | |
| *** gyee has joined #openstack-infra | 22:04 | |
| lifeless | clarkb: saturday morning? not hugely great. | 22:05 |
| lifeless | clarkb: Lynne is @ the gym, I have Cynthia for next 45m or so, then there is a slot where I can focus | 22:05 |
| lifeless | clarkb: or I can do the occasional drive by comment, like these. | 22:06 |
| clarkb | is it Friday already? | 22:08 |
| clarkb | lifeless: thats fine, I just remembered it was a thing that needed doing | 22:09 |
| *** sdake has quit IRC | 22:09 | |
| *** sdake has joined #openstack-infra | 22:10 | |
| *** pcm_ has quit IRC | 22:11 | |
| *** jcoufal has quit IRC | 22:12 | |
| *** sarob has joined #openstack-infra | 22:16 | |
| *** boris-42 has quit IRC | 22:21 | |
| *** NobodyCa1 has joined #openstack-infra | 22:23 | |
| *** NobodyCa1 has quit IRC | 22:24 | |
| *** sarob has quit IRC | 22:25 | |
| pleia2 | clarkb: I'm around, so we could sketch out some initial thoughts, or just wait until next week | 22:25 |
| *** zeus has quit IRC | 22:29 | |
| *** sarob has joined #openstack-infra | 22:29 | |
| *** NobodyCa1 has joined #openstack-infra | 22:31 | |
| *** jerryz has joined #openstack-infra | 22:33 | |
| *** sarob has quit IRC | 22:34 | |
| *** sarob has joined #openstack-infra | 22:35 | |
| *** NobodyCa1 has quit IRC | 22:36 | |
| clarkb | pleia2: now is goodish I need to push a change really quick but then will have time | 22:36 |
| pleia2 | k | 22:38 |
| *** NobodyCa1 has joined #openstack-infra | 22:40 | |
| *** NobodyCa1 has quit IRC | 22:41 | |
| clarkb | pleia2: I am in the etherpad, should we use chat there? | 22:42 |
| *** NobodyCam has joined #openstack-infra | 22:43 | |
| jog0 | clarkb: just when I think I get things working no tempest jobs fail, I don't know if Iam happy or sad | 22:45 |
| clarkb | jog0: ha | 22:45 |
| clarkb | jog0: you could be bad and push your own WIP tempest change that is intentionally broken >_> | 22:46 |
| jog0 | hmm thats not a bad idea | 22:47 |
| clarkb | jog0: as long as it is WIPed I think it is fine | 22:47 |
| jog0 | it won't test everything because all the logstash queries will be reuturn false | 22:47 |
| jog0 | yeah | 22:47 |
| fungi | i don't think there are enough people around approving changes right now who would care | 22:47 |
| pleia2 | clarkb: yeah | 22:48 |
| *** reed has joined #openstack-infra | 22:49 | |
| pleia2 | hm, toci results used to be reported here http://54.228.118.193/toci/ | 22:50 |
| pleia2 | I guess I missed the memo on when it moved/stopped | 22:50 |
| jog0 | if WIPed jenkins still runs it right | 22:51 |
| jog0 | https://review.openstack.org/46582 | 22:51 |
| clarkb | jog0: correct WIP runs tests Draft does not | 22:52 |
| jog0 | awesome | 22:52 |
| *** lnxnut has quit IRC | 22:52 | |
| *** wenlock has quit IRC | 22:59 | |
| clarkb | pleia2: I think it would be helpful to describe the process without using specifics like LXC or KVM | 22:59 |
| pleia2 | clarkb: yeah, good idea | 23:00 |
| pleia2 | oh, hah, nicklist in etherpad covered the chat, nice | 23:01 |
| fungi | UtahDave: playing around with the salt trigger slave... what should i be able to tack on to the end of 'sudo /usr/bin/salt-call event.fire_master' as the jenkins user there? | 23:01 |
| fungi | nevermind, figured it out | 23:04 |
| fungi | sudo /usr/bin/salt-call event.fire_master jenkins test.sls | 23:04 |
| fungi | local: | 23:04 |
| fungi | True | 23:04 |
| fungi | seems worky | 23:04 |
| jog0 | clarkb: is logstash rate limiting me? | 23:04 |
| jog0 | requests.exceptions.ConnectionError: HTTPConnectionPool(host='logstash.openstack.o | 23:05 |
| *** dmakogon_ has quit IRC | 23:06 | |
| clarkb | jog0: I don't think so | 23:06 |
| fungi | 'sudo salt-call event.fire_master jenkins test' also seems to work, ftr | 23:06 |
| clarkb | it does have timeouts though maybe you ran into one of those? | 23:07 |
| jog0 | something is timing out then | 23:07 |
| UtahDave | fungi: sudo /usr/bin/salt-call event.fire_master 'unused data text' 'jenkins' | 23:09 |
| clarkb | jog0: connectiontimeout is 5 seconds timeout is 120 seconds | 23:10 |
| clarkb | jog0: I bet you ran into one of those | 23:10 |
| clarkb | jog0: but I don't see any errors in the apache logs | 23:10 |
| UtahDave | fungi: the signature is salt.modules.event.fire_master(data, tag) | 23:11 |
| jog0 | logstash can be slow to search the first time | 23:11 |
| clarkb | jog0: yeah, it needs to warm its caches | 23:12 |
| jog0 | which means I timeout | 23:13 |
| jog0 | can we bump up the 5 second timeout? | 23:14 |
| clarkb | jog0: yes | 23:14 |
| *** UtahDave has quit IRC | 23:15 | |
| jog0 | oh so just classified my first bug! | 23:15 |
| jog0 | to bad Alex_Gaynor beat me to the recheck | 23:15 |
| Alex_Gaynor | jog0: sorry! | 23:15 |
| clarkb | jog0: nice! | 23:16 |
| clarkb | jog0: any idea what the increased timeout should be? is 15 seconds still going to be too low? | 23:16 |
| Alex_Gaynor | jog0: also: awesome! | 23:16 |
| jog0 | clarkb: not sure | 23:16 |
| jog0 | 15 why not | 23:17 |
| jog0 | lets start with taht | 23:17 |
| *** dkehn_ has joined #openstack-infra | 23:18 | |
| clarkb | we seem to have lost the gerritbot | 23:18 |
| clarkb | jog0: https://review.openstack.org/46584 I am going to apply that by hand for now (and disable puppet on the logstash.o.o host) | 23:18 |
| clarkb | fungi: ^ | 23:18 |
| clarkb | jog0: done | 23:19 |
| jog0 | https://github.com/jogo/elasticRecheck/blob/master/queries.json | 23:20 |
| jog0 | https://github.com/jogo/elasticRecheck/blob/master/tests/test_classifier.py#L19 | 23:20 |
| jog0 | got that test to pass :) | 23:20 |
| clarkb | jog0: \o/ | 23:21 |
| clarkb | jog0: note that we only keep 2 weeks of logs in elasticsearch so tests that hardcode values like that may not always work... | 23:22 |
| clarkb | jog0: might need to mock elasticsearch or bundle it or something | 23:22 |
| Alex_Gaynor | clarkb: how large is our elasticsearch cluster? | 23:22 |
| clarkb | Alex_Gaynor: 6 30G nodes | 23:22 |
| jog0 | clarkb: yeah I figured | 23:22 |
| clarkb | Alex_Gaynor: now that I stopped indexing debug logs it can probably increase the backlog | 23:22 |
| clarkb | Alex_Gaynor: but it was really strained when we indexed debug logs | 23:22 |
| jog0 | right now this is very much a POC and once its working all the way through we can fix things like that | 23:22 |
| clarkb | jog0: ++ | 23:23 |
| Alex_Gaynor | clarkb: sorry, I meant how big in terms of stored data | 23:23 |
| *** dkehn_ has quit IRC | 23:23 | |
| jog0 | I was thinking for now make this interface to a ircbot in #-qa | 23:23 |
| clarkb | Alex_Gaynor: oh, with debug logs it was ~5TB for the indexes (notes indexes are expensive compared to raw data), now we have about 14 * 2 * 25GB | 23:24 |
| jog0 | and eventually write a comment on failed jobs saying 'I think you hit x', and still do irc bot for unclassfied failures | 23:24 |
| clarkb | jog0: that makes sense to me | 23:24 |
| clarkb | Alex_Gaynor: we are a bit oversized right now, but I was always undersized in the past so am happy with the change | 23:25 |
| clarkb | Alex_Gaynor: looks like about 600GB of indexes | 23:25 |
| reed | pleia2, rememebr to send notification of events to events@openstack.org so I don't have to enter those :) | 23:30 |
| *** mrodden has quit IRC | 23:31 | |
| pleia2 | reed: oh! shall I do that now? | 23:31 |
| reed | pleia2, too late :) I'm copying already | 23:32 |
| pleia2 | ok | 23:32 |
| pleia2 | next time, I promise | 23:32 |
| reed | hopefully we'll have the group portal running soon and you'll be able to enter that stuff yourself there | 23:32 |
| pleia2 | yeah, that would be nice | 23:32 |
| reed | has that patch from mrmarting landed? | 23:32 |
| reed | ah, not yet... what a week | 23:33 |
| reed | ladies and gentlemen, what a week I say | 23:33 |
| pleia2 | clarkb: so on Monday the tripleo folks are going to do a tripleo overview, I'm thinking that will be very useful as we dive into discussing testing, there are portions of over/undercloud stuff that have gotten a bit away from me too (since that hasn't been my focus) | 23:34 |
| clarkb | pleia2: ok do you know what time that is? | 23:34 |
| clarkb | pleia2: I should be able to attend (mostly concerned about sleeping through it) | 23:34 |
| pleia2 | so it's not strictly on the schedule https://etherpad.openstack.org/tripleo-havana-sprint | 23:35 |
| pleia2 | but probably 1st thing on Monday | 23:35 |
| pleia2 | bootstrapping everyone will require explaining things :) | 23:35 |
| clarkb | ok I will do my best to get in early | 23:36 |
| clarkb | pleia2: there is a big football game on Sunday night :) | 23:37 |
| pleia2 | hehe | 23:37 |
| pleia2 | that's the game with the oval-ish ball | 23:37 |
| clarkb | yes made with pig skin | 23:37 |
| *** MarkAtwood has quit IRC | 23:38 | |
| jog0 | clarkb: so getting false positives because of the bug we talekd about before | 23:41 |
| *** fbo is now known as fbo_away | 23:42 | |
| clarkb | jog0: one way to mitigate that until I have a fix written is to check the timestamps of the log message you get back against the timestamp for the job completion event | 23:43 |
| *** sarob_ has joined #openstack-infra | 23:43 | |
| clarkb | jog0: that happens when the same change,patchset runs the tests multiple times so the timestamp should give you decent amount of info on whether or not the run you aredealing with is the one you have found in logstash | 23:43 |
| jog0 | hmm will check that out | 23:44 |
| clarkb | jog0: I will try and give the stuff more priority but next week is the tripleo sprint and I think we are going to try and get some work on testing tripleo | 23:44 |
| jog0 | clarkb: I will be there too | 23:44 |
| clarkb | oh awesome | 23:44 |
| jog0 | and in bend | 23:44 |
| clarkb | jog0: maybe we can take a half hour or so to go over what the setup looks like so you have some familiarity with the stuff behind the scenes too? | 23:44 |
| jog0 | sure | 23:45 |
| jog0 | clarkb: your saying its not magic? :( | 23:45 |
| *** hemna is now known as hemnafk | 23:45 | |
| clarkb | jog0: no unfortunately not | 23:45 |
| clarkb | if I get bored over the weekend I may try fixing that bug | 23:46 |
| jog0 | btw with the two changes, this looks much smaller http://status.openstack.org/rechecks/ | 23:46 |
| jog0 | just 74 bugs lol | 23:46 |
| clarkb | :) | 23:47 |
| *** sarob has quit IRC | 23:47 | |
| jog0 | and I need to go and mark dupes at some point | 23:47 |
| clarkb | jog0: pleia2: anteaya: when do you guys get to seattle? is Sunday lunch/dinner something that should happen? | 23:47 |
| *** sarob_ has quit IRC | 23:48 | |
| jog0 | sunday at 6ish | 23:49 |
| jog0 | so I can do dinner | 23:49 |
| pleia2 | clarkb: I'm wheels down at 19:53 on sunday, so probably just going to grab something quick after checking in at hotel | 23:49 |
| pleia2 | I think anteaya is arriving saturday though | 23:50 |
| jog0 | clarkb: ther is nothing I can do with the job number to sort out dupes right? | 23:51 |
| clarkb | jog0: not since we moved to multiple jenkins masters | 23:52 |
| clarkb | jog0: I mean you can approxmiate, but can't be sure | 23:52 |
| jog0 | I am afraid teh timestamp logic will be a little iffy too | 23:52 |
| clarkb | jog0: I think using the job numbers for now if probably going to get you close enough | 23:53 |
| clarkb | jog0: and should hold you over until the uuids are available | 23:53 |
| jog0 | how would i do that? | 23:53 |
| *** dripton has quit IRC | 23:53 | |
| clarkb | the logstash event has a build number compare that to the build number on the other end (gerrit ssh event stream?) | 23:53 |
| clarkb | oh except we don't put the build number in the gerrit comments anymore :( | 23:54 |
| jog0 | clarkb: yeah ... | 23:54 |
| jog0 | is it easier to just add them back? | 23:54 |
| clarkb | no, better to put uuids in the logstash events | 23:54 |
| clarkb | let me see if I can hack something up in 15 minutes | 23:55 |
| *** dripton has joined #openstack-infra | 23:57 | |
| pleia2 | I need to go prep for a pre-sunset dinner, then offline through tomorrow night | 23:57 |
| pleia2 | chat sunday, see many of you monday :) | 23:57 |
| jog0 | clarkb: I have a solution I think, its ugly but will work | 23:58 |
| jog0 | @fields.build_change:"45335" AND (@message:"Finished: SUCCESS" OR @message:"Finished: SUCCESS") AND @fields.build_patchset:"6" | 23:58 |
| jog0 | look through those results, as sorted by time and note the most recent build numbers | 23:58 |
| jog0 | pleia2: have an easy fast | 23:59 |
| pleia2 | jog0: thank you :) | 23:59 |
| *** lnxnut has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!