| anteaya | not in projects.yaml | 00:00 |
|---|---|---|
| anteaya | well first they have to go through jenkins job builder | 00:00 |
| anteaya | in order for the jobs to be build in xml | 00:00 |
| anteaya | and to be honest I don't think I have ever asked the exact question you are asking | 00:00 |
| JayF | The arrangement seems to be pseudo-arbitrary, but I'm sure there's some rhyme or reason to it | 00:01 |
| anteaya | but I believe that the way jenkins job builder builds the jobs it is configured to read certain kinds of information from certain yaml files | 00:01 |
| JayF | either way I think I have this patch fixed, we'll see ... | 00:01 |
| anteaya | okay | 00:01 |
| anteaya | the other thing is that projects.yaml is very very large | 00:01 |
| anteaya | it would be even larger if we also allowed templates to be defined inside of it too | 00:02 |
| *** julim has joined #openstack-infra | 00:02 | |
| JayF | yeah I think splitting more out of it would've made what I'm doing a lot easier | 00:02 |
| *** thorst_ has quit IRC | 00:02 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs https://review.openstack.org/313800 | 00:03 |
| anteaya | well there are an exceptional amount of jobs called on the ironic repo in projects.yaml | 00:03 |
| anteaya | most other projects dont' have nearly that many jobs on them | 00:04 |
| *** johnny___ has quit IRC | 00:04 | |
| anteaya | I don't know why tripleo jobs are defined on the ironic repo in projects.yaml | 00:04 |
| *** aeng has quit IRC | 00:04 | |
| *** camunoz has quit IRC | 00:04 | |
| anteaya | I need to stop talking about having something to eat and have something to eat | 00:05 |
| *** sflanigan has quit IRC | 00:05 | |
| anteaya | stepping away | 00:05 |
| *** pahuang has quit IRC | 00:05 | |
| JayF | same, ty for the help and would appreciate reviews on ^ | 00:06 |
| pabelanger | jeblair: I don't believe so. | 00:07 |
| jeblair | pabelanger: yah, i think we tracked it down -- seems we always needed 'admin' privs to delete | 00:07 |
| pabelanger | anteaya: JayF: yes, they should be moved into tripleo-ci project if people are discussion reordering | 00:08 |
| *** MarkAtwood has joined #openstack-infra | 00:08 | |
| pabelanger | anteaya: JayF: I can submit a follow up patch in the morning for your review if you'd like | 00:08 |
| pabelanger | jeblair: ack | 00:08 |
| *** ffrank has quit IRC | 00:09 | |
| *** sridhar_ram has quit IRC | 00:10 | |
| krotscheck | EmilienM: Sorry I wasn't around. My concern is about the actual mirror file structure on disk- AFS has a practical folder size limit of ~32K records, and if the mirror is structured to put all packages into one dir, that'll blow up on us. | 00:10 |
| *** kzaitsev_mb has joined #openstack-infra | 00:11 | |
| *** camunoz has joined #openstack-infra | 00:12 | |
| pabelanger | clarkb: ianw: I have deleted bare-precise from nodepool, we are ready to land https://review.openstack.org/#/c/313631/ | 00:14 |
| *** ddieterly has joined #openstack-infra | 00:16 | |
| *** grauzikas has quit IRC | 00:17 | |
| *** IlyaG has quit IRC | 00:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Remove bare-precise from nodepool-builder logging https://review.openstack.org/314812 | 00:17 |
| *** kzaitsev_mb has quit IRC | 00:17 | |
| pabelanger | and updated logging for nodepool-builder ^ | 00:17 |
| *** sflanigan has joined #openstack-infra | 00:18 | |
| *** aeng has joined #openstack-infra | 00:20 | |
| ianw | pabelanger: ok, cool. probably someone who can log in should do final approval. | 00:22 |
| *** chlong_ has joined #openstack-infra | 00:22 | |
| *** chlong_ has quit IRC | 00:22 | |
| *** pahuang has joined #openstack-infra | 00:22 | |
| ianw | does that mean we are officially off snapshots everywhere? | 00:22 |
| pabelanger | ianw: everywhere minus tripleo | 00:22 |
| pabelanger | ianw: however I have a patch to fix that https://review.openstack.org/#/c/312725/ | 00:23 |
| pabelanger | hoping to land that this week | 00:23 |
| pabelanger | then plan the migration away from fedora-22 next | 00:23 |
| pabelanger | ianw: then we can close out: https://specs.openstack.org/openstack-infra/infra-specs/specs/dib-nodepool.html | 00:23 |
| *** jamesmca_ has joined #openstack-infra | 00:26 | |
| *** chlong_ has joined #openstack-infra | 00:27 | |
| *** salv-orlando has joined #openstack-infra | 00:28 | |
| *** chlong_ has quit IRC | 00:28 | |
| *** Swami has quit IRC | 00:29 | |
| *** jamesmca_ has quit IRC | 00:30 | |
| anteaya | pabelanger: thanks yes, I think having tripleo jobs defined on tripleo repos would help avoid future confusion | 00:30 |
| *** salv-orlando has quit IRC | 00:33 | |
| tonyb | fungi: what is the process for requesting kilo-eol tags in the projects are are now EOLd? | 00:34 |
| anteaya | hello tonyb | 00:36 |
| tonyb | anteaya: hey there! | 00:37 |
| anteaya | tonyb: there you are being stable again | 00:37 |
| anteaya | I just love that you're the stable one | 00:37 |
| anteaya | ha ha ha | 00:37 |
| tonyb | anteaya: I do try to hold up my end of the bargin from time to time :) | 00:37 |
| anteaya | you do a great job | 00:37 |
| fungi | sdake: i'm around again for a few minutes, but protracted discussion should probably wait for tomorrow | 00:38 |
| fungi | tonyb: first make sure there are no open changes for those branches | 00:38 |
| tonyb | fungi: cool I can do that | 00:38 |
| ianw | pabelanger: 2 years, 7 months, 28 days since https://review.openstack.org/#/c/46482/ . | 00:38 |
| *** amitgandhinz has joined #openstack-infra | 00:39 | |
| fungi | tonyb: abandon any you find with a consistent message about the branch going away | 00:39 |
| fungi | tonyb: then push a "kilo-eol" tag to the stable/kilo branches of any projects under official stable branch management | 00:39 |
| pabelanger | ianw: Ya, I'm pretty happy how well its been coming together over the last few weeks | 00:40 |
| tonyb | fungi: Oh I can do that? neat. | 00:40 |
| fungi | tonyb: or get a gerrit admin to do it. i've been doing it since we eol'd diablo i think | 00:40 |
| fungi | you might not have tag permissions on them | 00:41 |
| pabelanger | fungi: mind doing the honors on: https://review.openstack.org/#/c/313631/ ? I can restart all the nodepool bits | 00:41 |
| fungi | tonyb: anyway, he biggest help will be tracking down open changes and putting together the list of repos which need kilo-eol treatment | 00:41 |
| tonyb | fungi: okay it looks like I don't have tag permission so I'll do the other work and then email openstack-infra with the request. | 00:43 |
| fungi | pabelanger: no logging config adjustment needed to go along with that? | 00:43 |
| tonyb | fungi: Thanks for putting me on the right path. | 00:43 |
| fungi | tonyb: sounds good. i'm happy to continue doing that piece. it's easy to do a shell loop one-liner | 00:43 |
| *** hichihara has joined #openstack-infra | 00:44 | |
| pabelanger | fungi: we do! https://review.openstack.org/#/c/314812/ | 00:44 |
| tonyb | fungi: cool. | 00:44 |
| fungi | pabelanger: cool, all lgtm. thanks for working on that! | 00:45 |
| pabelanger | fungi: Ya, I'll very pleased once we move tripleo-ci to a DIB | 00:46 |
| *** jpr has quit IRC | 00:47 | |
| *** mtanino has quit IRC | 00:48 | |
| *** timl0101 has quit IRC | 00:49 | |
| *** asselin has joined #openstack-infra | 00:53 | |
| *** bpokorny_ has joined #openstack-infra | 00:56 | |
| *** madhu_ak has quit IRC | 00:56 | |
| *** bpokorny has quit IRC | 01:00 | |
| *** bpokorny_ has quit IRC | 01:01 | |
| *** crinkle has joined #openstack-infra | 01:03 | |
| *** esikachev has joined #openstack-infra | 01:03 | |
| anteaya | found out my osteopath who got sick summer of 2014 is alive!! | 01:03 |
| *** yamamot__ has joined #openstack-infra | 01:04 | |
| anteaya | finally so good to know something about her | 01:04 |
| anteaya | and it is good news!! | 01:04 |
| anteaya | happy happy happy happy | 01:04 |
| *** esikachev has quit IRC | 01:08 | |
| *** tphummel has quit IRC | 01:10 | |
| *** zhurong has joined #openstack-infra | 01:11 | |
| *** pvaneck has quit IRC | 01:11 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:12 | |
| *** ramishra_ has joined #openstack-infra | 01:12 | |
| *** tqtran has quit IRC | 01:12 | |
| fungi | guess we're going to have to stop avoiding toml soon... https://mail.python.org/pipermail/distutils-sig/2016-May/028889.html | 01:14 |
| dstufft | why were you avoiding toml | 01:15 |
| fungi | some of us were turned off by the author basically stating that he wrote toml because he didn't feel like reading the entire yaml spec | 01:16 |
| fungi | which seemed like a terrible reason to go invent yet another serialization standard | 01:16 |
| dstufft | fungi: it is not the greatest reason to make yet another thing. | 01:18 |
| jeblair | that pep seems to echo that meme :( | 01:18 |
| *** kzaitsev_mb has quit IRC | 01:18 | |
| *** asselin has quit IRC | 01:19 | |
| jeblair | "86 pages if printed on letter-sized paper" | 01:19 |
| dstufft | I mean, the spec is long, and full of bits people rarely use. Conceptual overhead is a thing | 01:19 |
| fungi | i bet if i printed it double-spaced it would be even longer | 01:19 |
| jeblair | fungi: and in large type (though the 86 page thing did not specify type size or margins) | 01:19 |
| *** mixos has joined #openstack-infra | 01:20 | |
| jeblair | it also ends with a note about how yaml could be used in a way that addresses concerns about it. and then ends. | 01:21 |
| fungi | though i confirm, having more than once tried to figure out how to sanely overload pyyaml to raise exceptions on duplicate keys, that its particular parser implementation is a bit of a bear | 01:21 |
| dstufft | jeblair: that was basically, I spent a few hours today ripping out the things in pyyaml that make it really hard to vendor inside of pip | 01:21 |
| openstackgerrit | liusheng proposed openstack/requirements: bump aodhclient upper-constraints to version 0.5.0 https://review.openstack.org/314822 | 01:21 |
| fungi | and i can understand not waning to vendor pyyaml inside pip | 01:21 |
| dstufft | but like, I really really don't want to maintain a yaml parser | 01:21 |
| fungi | er, wanting | 01:22 |
| dstufft | like really really | 01:22 |
| pabelanger | So, does anybody have any thoughts to the following dependency issue with pypi? | 01:24 |
| jeblair | dstufft: well, thanks for doing that, it seems useful. i'm sorry to hear that it won't be used | 01:24 |
| pabelanger | http://logs.openstack.org/31/313631/3/check/gate-project-config-layout/4fbf5c0/console.html#_2016-05-11_01_15_48_375 | 01:24 |
| fungi | for the use case in that pep, i have a feeling the toml config representation will basically look about the same as if it had just used configparser anyway | 01:24 |
| pabelanger | I'm guessing there is an issue with rax-iad mirror? | 01:24 |
| *** kengo_sakai has joined #openstack-infra | 01:24 | |
| pabelanger | or maybe because we are still using the read write mirror and don't have atomic updates? | 01:25 |
| jeblair | pabelanger: we're on read-write until https://review.openstack.org/314800 lands | 01:25 |
| dstufft | jeblair: It's on github if anyone wants it :] (https://github.com/dstufft/yaml) I might finish fleshing it out with the other things I was going to change in it if I get bored | 01:25 |
| jeblair | dstufft: cool :) | 01:26 |
| pabelanger | jeblair: why yes, let me +A | 01:26 |
| *** piet has joined #openstack-infra | 01:28 | |
| jeblair | pabelanger: however, the upshot of that is that we should be even more up to date than normal | 01:28 |
| dstufft | FWIW I think it's a bit silly to focus on the "N pages long!!!" aspect of things, but I think it's a real concern that YAML is a fairly lengthy spec that contains a lot of things that most people will just flat out never need. Now it's true that if you don't need X feature, you don't have to use X feature, but you still tend to end up coming across people using all the little corners of it and having to know them to interact with | 01:29 |
| dstufft | other people's projects | 01:29 |
| pabelanger | jeblair: great | 01:29 |
| *** amotoki has quit IRC | 01:30 | |
| fungi | dstufft: that stripped-down (i prefer using its pure python implementation anyway) and modernized pyyaml fork looks useful | 01:30 |
| jeblair | yeah; if we add an option to detect duplicate keys, we could have a winner :) | 01:31 |
| dstufft | it has some code to do that already I think | 01:31 |
| dstufft | it's commented out though | 01:31 |
| jeblair | (iirc, the pyyaml maintainers don't want to add that because it's so 'easy' to extend the parser) | 01:31 |
| fungi | well, also it had something to do with the yaml spec not forbidding duplicate keys (which results in an undefined behavior, so arguably worse) | 01:32 |
| *** ramishra_ has quit IRC | 01:32 | |
| jeblair | yeah, but making it an _option_ is a good idea, methinks | 01:32 |
| dstufft | https://github.com/dstufft/yaml/blob/master/src/yaml/composer.py#L132-L138 | 01:33 |
| jeblair | pabelanger: so i don't know why that error happened; satisfactory files exist... could try checking the apache logs maybe | 01:34 |
| fungi | dstufft: wow, it really is just in there and commented out | 01:34 |
| dstufft | Note: I'm not 100% sure what the "Composer" file actually does | 01:35 |
| dstufft | the lay out of the code is confusing | 01:35 |
| dstufft | but that seems to do what you're looking for anyways | 01:35 |
| fungi | indeed, aside from similarly not wanting to maintain my own yaml parser ;) | 01:36 |
| *** mjblack has joined #openstack-infra | 01:36 | |
| fungi | but yeah, i'm curious to uncomment and try that out sometime when i'm not about to turn in for the night | 01:36 |
| pabelanger | jeblair: Ya, I might dig into it in the morning, see what happened | 01:36 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Start using read/write AFS volume for PyPI mirror" https://review.openstack.org/314800 | 01:37 |
| *** yamamot__ has quit IRC | 01:40 | |
| *** amotoki has joined #openstack-infra | 01:41 | |
| *** yamamot__ has joined #openstack-infra | 01:41 | |
| *** mriedem has quit IRC | 01:45 | |
| anteaya | we have the sprint channel for infra trusty upgrade: https://wiki.openstack.org/wiki/VirtualSprints#Infra_Trusty_Upgrade | 01:49 |
| *** yamahata has quit IRC | 01:49 | |
| anteaya | fungi: I listed you as the event contact for that | 01:49 |
| *** amotoki has quit IRC | 01:50 | |
| *** timl0101 has joined #openstack-infra | 01:51 | |
| *** tphummel has joined #openstack-infra | 01:51 | |
| *** timl0101 has quit IRC | 01:52 | |
| *** thorst_ has joined #openstack-infra | 01:53 | |
| *** piet has quit IRC | 01:54 | |
| *** anteaya has quit IRC | 01:54 | |
| *** timl0101 has joined #openstack-infra | 01:55 | |
| *** thorst__ has joined #openstack-infra | 01:55 | |
| *** amotoki has joined #openstack-infra | 01:56 | |
| *** thorst___ has joined #openstack-infra | 01:57 | |
| *** thorst___ has quit IRC | 01:57 | |
| *** thorst___ has joined #openstack-infra | 01:57 | |
| *** thorst_ has quit IRC | 01:57 | |
| EmilienM | krotscheck: maybe we need a second dir? | 01:58 |
| *** salv-orlando has joined #openstack-infra | 01:58 | |
| EmilienM | krotscheck: is it useful if I tell how ~ gems we would have? | 01:58 |
| *** thorst__ has quit IRC | 01:59 | |
| *** DevBox has joined #openstack-infra | 02:03 | |
| *** jpr has joined #openstack-infra | 02:04 | |
| *** salv-orlando has quit IRC | 02:05 | |
| *** kzaitsev_mb has joined #openstack-infra | 02:14 | |
| *** ddieterly is now known as ddieterly[away] | 02:15 | |
| *** hdd has joined #openstack-infra | 02:17 | |
| *** amotoki has quit IRC | 02:19 | |
| *** gyee has quit IRC | 02:20 | |
| *** kzaitsev_mb has quit IRC | 02:20 | |
| *** jianghuaw has quit IRC | 02:24 | |
| *** fitoduarte has quit IRC | 02:26 | |
| *** jamesmca_ has joined #openstack-infra | 02:28 | |
| *** tlian has quit IRC | 02:28 | |
| *** yamahata has joined #openstack-infra | 02:29 | |
| *** winggundamth has quit IRC | 02:29 | |
| openstackgerrit | Lana Brindley proposed openstack-infra/irc-meetings: Updating Install Guide meeting details https://review.openstack.org/314831 | 02:29 |
| openstackgerrit | Qiming Teng proposed openstack-infra/project-config: api-ref gate for senlin https://review.openstack.org/314832 | 02:30 |
| *** cody-somerville has joined #openstack-infra | 02:31 | |
| *** cody-somerville has quit IRC | 02:31 | |
| *** cody-somerville has joined #openstack-infra | 02:31 | |
| *** jpr has quit IRC | 02:32 | |
| *** amotoki has joined #openstack-infra | 02:32 | |
| openstackgerrit | Qiming Teng proposed openstack-infra/project-config: api-ref gate for senlin https://review.openstack.org/314832 | 02:32 |
| *** jamesmca_ has quit IRC | 02:32 | |
| *** dingyichen has joined #openstack-infra | 02:33 | |
| *** zhurong has quit IRC | 02:36 | |
| *** zhurong has joined #openstack-infra | 02:36 | |
| *** baoli has joined #openstack-infra | 02:38 | |
| *** yamamot__ has quit IRC | 02:38 | |
| *** aeng has quit IRC | 02:44 | |
| *** camunoz has quit IRC | 02:45 | |
| *** sflanigan has quit IRC | 02:45 | |
| *** piet has joined #openstack-infra | 02:45 | |
| *** tphummel has quit IRC | 02:46 | |
| stevemar | looks like a bunch of jobs just failed :\ | 02:53 |
| stevemar | http://logs.openstack.org/86/255686/8/gate/gate-keystone-python34-db/cbfa7fb/console.html#_2016-05-11_01_30_18_818 | 02:53 |
| stevemar | No matching distribution found for oslo.messaging===5.0.0 | 02:53 |
| stevemar | hmmm | 02:54 |
| stevemar | http://logs.openstack.org/42/222042/5/gate/gate-keystonemiddleware-python34/19a3eea/console.html#_2016-05-11_01_27_23_535 | 02:54 |
| stevemar | No matching distribution found for oslo.messaging>=4.5.0 | 02:54 |
| *** aeng has joined #openstack-infra | 02:56 | |
| *** sflanigan has joined #openstack-infra | 02:57 | |
| *** camunoz has joined #openstack-infra | 02:57 | |
| *** apoorvad has quit IRC | 02:58 | |
| *** tphummel has joined #openstack-infra | 03:00 | |
| *** binbincong has quit IRC | 03:00 | |
| *** Daisy has joined #openstack-infra | 03:03 | |
| jamielennox | stevemar: they are all py3 jobs - and looking at https://pypi.python.org/pypi/oslo.messaging it has py version == py2 | 03:04 |
| jamielennox | maybe? | 03:04 |
| *** ccamacho has quit IRC | 03:04 | |
| *** esikachev has joined #openstack-infra | 03:04 | |
| stevemar | jamielennox: maybe? | 03:04 |
| jamielennox | stevemar: it's a guess | 03:04 |
| stevemar | jamielennox: it was released yesterday, weird that it would start today | 03:05 |
| dstufft | sigh | 03:05 |
| jamielennox | but it looks like it's not packaged for 3 | 03:05 |
| dstufft | It's pip's fault. | 03:05 |
| stevemar | jamielennox: maybe http://logs.openstack.org/79/314779/2/check/gate-releases-pep8/41f4e0a/console.html#_2016-05-11_02_31_46_461 | 03:05 |
| stevemar | No matching distribution found for lazr.restfulclient>=0.9.19 (from launchpadlib->releases==0.0.1.dev1488) | 03:05 |
| stevemar | looks like it's not just oslo.messaging | 03:05 |
| *** ccamacho has joined #openstack-infra | 03:05 | |
| stevemar | dstufft: whats the dealio? | 03:05 |
| *** dimtruck is now known as zz_dimtruck | 03:06 | |
| *** zz_dimtruck is now known as dimtruck | 03:06 | |
| jamielennox | oh, then i've no idea :) | 03:06 |
| *** piet has quit IRC | 03:07 | |
| *** Daisy has quit IRC | 03:08 | |
| *** esikachev has quit IRC | 03:09 | |
| *** binbincong has joined #openstack-infra | 03:12 | |
| *** amitgandhinz has quit IRC | 03:12 | |
| *** jamesmca_ has joined #openstack-infra | 03:12 | |
| dstufft | stevemar: pip 8.1.2 (just released) pulls in a newer version of setuptools which fully normalizes names as per PEP 503 | 03:13 |
| dstufft | and apparently bandersnatch (and others) don't handle that correctly when there is a . in the name | 03:13 |
| dstufft | it won't affect PyPI, but mirrors it will. | 03:14 |
| dstufft | and I think python-pip on Debian and ubuntu are going to have the same issue | 03:14 |
| *** ddieterly[away] has quit IRC | 03:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:17 | |
| *** jamesmca_ has quit IRC | 03:17 | |
| *** Daisy has joined #openstack-infra | 03:18 | |
| stevemar | dstufft: ouch, i guess we have to cap the pip version we use for now? hopefully someone from infra can solve this, i'm just the messenger :) | 03:19 |
| dstufft | I'm thinking. | 03:19 |
| dstufft | we upgraded setuptools because of $ISSUES | 03:19 |
| dstufft | (the one we bundle in pip) | 03:19 |
| dstufft | but that pulls in this change too | 03:19 |
| *** yamamot__ has joined #openstack-infra | 03:19 | |
| dstufft | (which this change is correct... I thought we already made it but apparently we missed a spot) | 03:20 |
| *** jianghuaw has joined #openstack-infra | 03:20 | |
| jamielennox | depends, if pip suddenly can't handle a . in a project name that will probably need to be fixed in a wider scope than us capping the version | 03:20 |
| *** pahuang has quit IRC | 03:20 | |
| jianghuaw | hi all. Is this a bug with nodepool: Per the nodepool doc, floating ip pool is supported. But it seems this parameter doesn’t take effective anymore. | 03:20 |
| jianghuaw | http://docs.openstack.org/infra/nodepool/configuration.html | 03:20 |
| jamielennox | oh, not pip, bandersnatch | 03:20 |
| jianghuaw | The ‘pool’ is generated from the configuration, but it is not used anywhere in the nodepool. https://github.com/openstack-infra/nodepool/blob/master/nodepool/config.py#L167 | 03:21 |
| openstackgerrit | Fei Long Wang proposed openstack-infra/irc-meetings: Propose changing Zaqar meeting time https://review.openstack.org/314848 | 03:21 |
| *** links has joined #openstack-infra | 03:21 | |
| dstufft | https://bitbucket.org/pypa/bandersnatch/pull-requests/20/fully-implement-pep-503-normalization/diff fixes it in bandersnatch | 03:21 |
| *** hdd has quit IRC | 03:22 | |
| jianghuaw | Per my understanding: The function supporting the floating ip pool was previously in nodepool; but the relative functions had been moved to shade and then the following patch changed nodepool to use shade for all Openstack interactions. In this commit, the floating pool function was removed from nodepool but was not switched to use shade even though which supports the concept of ip_pool indeed. https://review.openstack.or | 03:22 |
| *** yamamot__ has quit IRC | 03:22 | |
| *** kzaitsev_mb has quit IRC | 03:23 | |
| *** yamamot__ has joined #openstack-infra | 03:23 | |
| *** mjblack has quit IRC | 03:24 | |
| jianghuaw | I think we should change nodepool to pass the floating ip "pool" to shade APIs to support this documented function. Any ideas? | 03:24 |
| *** salv-orlando has joined #openstack-infra | 03:24 | |
| *** mjblack has joined #openstack-infra | 03:24 | |
| *** Daisy_ has joined #openstack-infra | 03:24 | |
| *** yamamot__ has quit IRC | 03:25 | |
| *** Daisy__ has joined #openstack-infra | 03:25 | |
| *** signed8bit_Zzz is now known as signed8bit | 03:26 | |
| *** armax has quit IRC | 03:27 | |
| *** yamamot__ has joined #openstack-infra | 03:27 | |
| *** Daisy has quit IRC | 03:27 | |
| *** Daisy_ has quit IRC | 03:29 | |
| *** salv-orlando has quit IRC | 03:30 | |
| *** pahuang has joined #openstack-infra | 03:31 | |
| openstackgerrit | Qiming Teng proposed openstack-infra/project-config: api-ref gate for senlin https://review.openstack.org/314832 | 03:31 |
| *** signed8bit has quit IRC | 03:32 | |
| *** Daisy__ has quit IRC | 03:34 | |
| *** Jeffrey4l has joined #openstack-infra | 03:35 | |
| *** pahuang has quit IRC | 03:35 | |
| *** salv-orlando has joined #openstack-infra | 03:36 | |
| *** binbincong has quit IRC | 03:36 | |
| *** hichihara has quit IRC | 03:36 | |
| *** tphummel has quit IRC | 03:36 | |
| *** camunoz has quit IRC | 03:39 | |
| ianw | dstufft: so you're saying that bandersnatch has created a broken mirror? that's why stuff like http://logs.openstack.org/31/313631/3/check/gate-project-config-layout/25464b3/console.html is failing on "oslo.config" (due to .)? | 03:39 |
| *** salv-orlando has quit IRC | 03:41 | |
| *** camunoz has joined #openstack-infra | 03:41 | |
| *** sree has joined #openstack-infra | 03:42 | |
| *** sree has joined #openstack-infra | 03:43 | |
| *** amitgandhinz has joined #openstack-infra | 03:43 | |
| *** aeng_ has joined #openstack-infra | 03:46 | |
| *** amitgandhinz has quit IRC | 03:48 | |
| *** aeng has quit IRC | 03:49 | |
| *** sflanigan has quit IRC | 03:49 | |
| *** camunoz has quit IRC | 03:50 | |
| *** gildub has joined #openstack-infra | 03:52 | |
| *** hdd has joined #openstack-infra | 03:53 | |
| *** fedexo has joined #openstack-infra | 03:55 | |
| *** binbincong has joined #openstack-infra | 03:59 | |
| *** hongbin has joined #openstack-infra | 04:00 | |
| *** hongbin has quit IRC | 04:01 | |
| *** mjblack has quit IRC | 04:01 | |
| *** baoli has quit IRC | 04:01 | |
| *** sflanigan has joined #openstack-infra | 04:02 | |
| *** pcaruana has joined #openstack-infra | 04:02 | |
| *** camunoz has joined #openstack-infra | 04:03 | |
| *** cody-somerville has quit IRC | 04:04 | |
| *** Qiming has quit IRC | 04:09 | |
| *** amitgandhinz has joined #openstack-infra | 04:14 | |
| *** weshay has quit IRC | 04:16 | |
| *** mjblack has joined #openstack-infra | 04:16 | |
| *** tphummel has joined #openstack-infra | 04:19 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:20 | |
| *** jamesmca_ has joined #openstack-infra | 04:23 | |
| *** esker has joined #openstack-infra | 04:24 | |
| *** kzaitsev_mb has quit IRC | 04:26 | |
| *** esker has quit IRC | 04:28 | |
| *** jamesmca_ has quit IRC | 04:28 | |
| *** timl0101 has quit IRC | 04:30 | |
| *** rfolco has quit IRC | 04:30 | |
| *** amitgandhinz has quit IRC | 04:42 | |
| *** psachin has joined #openstack-infra | 04:42 | |
| *** yuanying has quit IRC | 04:44 | |
| *** Qiming has joined #openstack-infra | 04:46 | |
| *** amotoki_ has joined #openstack-infra | 04:46 | |
| *** mrunge has quit IRC | 04:46 | |
| *** salv-orlando has joined #openstack-infra | 04:47 | |
| *** claudiub has joined #openstack-infra | 04:48 | |
| *** amotoki has quit IRC | 04:48 | |
| *** markvoelker_ has joined #openstack-infra | 04:50 | |
| *** salv-orlando has quit IRC | 04:52 | |
| *** javeriak has joined #openstack-infra | 04:53 | |
| *** dimtruck is now known as zz_dimtruck | 04:57 | |
| *** zz_dimtruck is now known as dimtruck | 04:57 | |
| *** rcernin has joined #openstack-infra | 04:58 | |
| *** icey has quit IRC | 05:02 | |
| *** rlandy has quit IRC | 05:03 | |
| *** jaosorior has joined #openstack-infra | 05:04 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Procedure to build UEFI based images https://review.openstack.org/307655 | 05:04 |
| *** fawadkhaliq has joined #openstack-infra | 05:05 | |
| *** esikachev has joined #openstack-infra | 05:05 | |
| *** dimtruck is now known as zz_dimtruck | 05:07 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 05:08 |
| *** yamamot__ has quit IRC | 05:08 | |
| *** sdake has quit IRC | 05:08 | |
| *** yamamot__ has joined #openstack-infra | 05:09 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:09 | |
| *** amitgandhinz has joined #openstack-infra | 05:09 | |
| *** esikachev has quit IRC | 05:10 | |
| *** yamamot__ has quit IRC | 05:12 | |
| *** javeriak has quit IRC | 05:12 | |
| *** yamamot__ has joined #openstack-infra | 05:13 | |
| *** savihou has joined #openstack-infra | 05:13 | |
| *** kzaitsev_mb has quit IRC | 05:14 | |
| *** fitoduarte has joined #openstack-infra | 05:14 | |
| *** abregman|afk is now known as abregman | 05:14 | |
| *** gildub has quit IRC | 05:22 | |
| *** roxanaghe has joined #openstack-infra | 05:25 | |
| *** salv-orlando has joined #openstack-infra | 05:28 | |
| *** javeriak has joined #openstack-infra | 05:31 | |
| *** shashank_hegde has joined #openstack-infra | 05:31 | |
| *** gomarivera has joined #openstack-infra | 05:33 | |
| *** camunoz has quit IRC | 05:33 | |
| *** hdd has quit IRC | 05:33 | |
| *** nadya has quit IRC | 05:38 | |
| *** roxanaghe has quit IRC | 05:39 | |
| *** javeriak has quit IRC | 05:41 | |
| *** burgerk has quit IRC | 05:41 | |
| *** fitoduarte has quit IRC | 05:42 | |
| *** burgerk has joined #openstack-infra | 05:42 | |
| *** javeriak has joined #openstack-infra | 05:47 | |
| *** roxanaghe has joined #openstack-infra | 05:48 | |
| *** nadya has joined #openstack-infra | 05:49 | |
| dstufft | ianw: sort of, the /simple/ API wasn't fully defined prior to PEP 503 and was super implenation specific. PEP 503 defines it but as part of that we had to make some decisions around which implementation's behavior we defined as standard. Bandersnatch attmpted to implement PEP 503 (via me doing it) and mistakenly missed an edge case in the normalization (that pio also happened to miss in some scenarios--- but which didn't matter | 05:50 |
| dstufft | because PyPI redirected) and the last N released of setuptools implemented things as PEP 503 said to do, so pip pulled that in with 8.1.2 as part of updating our bundled deps | 05:50 |
| *** vhosakot has quit IRC | 05:50 | |
| *** yamamot__ has quit IRC | 05:50 | |
| dstufft | ianw: so there isn't a great fix here, pip could roll back the setuptools upgrade, but we'll need to do it at some point and given it only affects mirrors, and people are going to need to upgrade their bandersnatches in the next month or two ish anyways, seems reasonable to push it out that direction | 05:51 |
| dstufft | anyways, bed -> | 05:51 |
| *** yamamot__ has joined #openstack-infra | 05:52 | |
| *** gomarivera has quit IRC | 05:52 | |
| *** nmagnezi has joined #openstack-infra | 05:52 | |
| *** sflanigan has quit IRC | 05:52 | |
| *** dchen has joined #openstack-infra | 05:53 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack https://review.openstack.org/309383 | 05:54 |
| *** amotoki_ has quit IRC | 05:56 | |
| *** roxanaghe has quit IRC | 05:57 | |
| *** javeriak has quit IRC | 05:57 | |
| *** roxanaghe has joined #openstack-infra | 05:57 | |
| *** zz_dimtruck is now known as dimtruck | 05:57 | |
| *** amitgandhinz has quit IRC | 06:00 | |
| *** roxanaghe has quit IRC | 06:01 | |
| *** yuanying has joined #openstack-infra | 06:01 | |
| *** pahuang_ has joined #openstack-infra | 06:02 | |
| *** gildub has joined #openstack-infra | 06:04 | |
| *** tphummel has quit IRC | 06:04 | |
| *** dingyichen has quit IRC | 06:05 | |
| *** dchen has quit IRC | 06:05 | |
| *** dingyichen has joined #openstack-infra | 06:06 | |
| *** hdaniel_ has joined #openstack-infra | 06:07 | |
| *** piet has joined #openstack-infra | 06:16 | |
| *** nadya has quit IRC | 06:18 | |
| *** mikelk has joined #openstack-infra | 06:19 | |
| *** david-lyle has quit IRC | 06:23 | |
| *** dimtruck is now known as zz_dimtruck | 06:24 | |
| *** piet has quit IRC | 06:25 | |
| zqfan | No matching distribution found for oslo.messaging>=4.5.0, this issue blocks all CI, I think , http://logs.openstack.org/02/314402/3/check/gate-ceilometer-python34/252179c/console.html#_2016-05-11_03_29_00_612 | 06:26 |
| *** aeng_ has quit IRC | 06:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/314878 | 06:26 |
| *** david-lyle has joined #openstack-infra | 06:26 | |
| *** amitgandhinz has joined #openstack-infra | 06:27 | |
| zqfan | it seems something wrong with the pypi mirror CI used | 06:29 |
| zqfan | http://mirror.regionone.bluebox-sjc1.openstack.org/pypi/packages/source/o/oslo.messaging/ | 06:29 |
| *** zz_dimtruck is now known as dimtruck | 06:30 | |
| *** javeriak has joined #openstack-infra | 06:34 | |
| *** shashank_hegde has quit IRC | 06:36 | |
| *** camunoz has joined #openstack-infra | 06:37 | |
| *** woodster_ has quit IRC | 06:38 | |
| skraynev | zqfan: +1 . I have the same issue, but with another requirements | 06:38 |
| skraynev | AJaeger: hi. could you help or redirect on right person about mentioned issue above ^ | 06:39 |
| *** aeng_ has joined #openstack-infra | 06:40 | |
| *** oanson has joined #openstack-infra | 06:42 | |
| rakhmerov | hi, same is happening to me | 06:42 |
| *** fawadkhaliq has quit IRC | 06:44 | |
| *** fawadkhaliq has joined #openstack-infra | 06:44 | |
| AJaeger | skraynev, zqfan: We need an infra-root to look at this. | 06:45 |
| AJaeger | jhesketh, are you around to help? | 06:46 |
| jhesketh | yep, just reading scrollback | 06:46 |
| AJaeger | thanks, jhesketh ! | 06:46 |
| skraynev | AJaeger: thank you. | 06:46 |
| *** fedexo has quit IRC | 06:47 | |
| *** dimtruck is now known as zz_dimtruck | 06:49 | |
| *** mixos has quit IRC | 06:49 | |
| *** MarkAtwood has quit IRC | 06:52 | |
| AJaeger | jhesketh, skraynev, zqfan : ianw did some analysis: http://lists.openstack.org/pipermail/openstack-dev/2016-May/094588.html | 06:54 |
| *** vgridnev has quit IRC | 06:54 | |
| jhesketh | AJaeger: oh interesting | 06:54 |
| openstackgerrit | Qiming Teng proposed openstack-infra/project-config: api-ref gate for senlin https://review.openstack.org/314832 | 06:54 |
| AJaeger | jhesketh: that looks like there's no way to fix it right now ;( | 06:54 |
| jhesketh | fwiw the oslo packages do exist in the simple mirror http://mirror.regionone.bluebox-sjc1.openstack.org/pypi/simple/oslo.messaging/ | 06:55 |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/zuul: Set keepalives for gerrit connections https://review.openstack.org/238988 | 06:55 |
| AJaeger | dstufft and ianw discussed this above ^ | 06:55 |
| *** vgridnev has joined #openstack-infra | 06:56 | |
| AJaeger | jhesketh: at 03:21 there's a change for bandersnatch... | 06:57 |
| rakhmerov | ooh.. | 06:57 |
| *** andreas_s has joined #openstack-infra | 06:57 | |
| AJaeger | So, pinning pip to < 8.1.2 should be a temporary fix. | 06:58 |
| jhesketh | AJaeger: are you referring to https://bitbucket.org/pypa/bandersnatch/pull-requests/20/fully-implement-pep-503-normalization/diff ? | 06:59 |
| AJaeger | jhesketh: yes - and looking at ianw's comment, it's not complete ;( | 06:59 |
| jhesketh | hmm I think a pin is a good idea | 06:59 |
| AJaeger | jhesketh: backscroll starts at 2:53 with StevenK mentioning it - and then see discussion by dstufft and ianw. | 07:00 |
| AJaeger | jhesketh: Unfortunately I cannot help you any further on this. | 07:00 |
| *** zz_dimtruck is now known as dimtruck | 07:01 | |
| *** salv-orlando has quit IRC | 07:02 | |
| AJaeger | jhesketh: I suggest that you give a #status alert pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved | 07:02 |
| AJaeger | Or something like that... | 07:02 |
| jhesketh | thanks, will do that as is | 07:03 |
| jhesketh | #status alert pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved | 07:03 |
| openstackstatus | jhesketh: sending alert | 07:03 |
| *** amotoki has joined #openstack-infra | 07:04 | |
| *** liusheng has joined #openstack-infra | 07:04 | |
| *** matrohon has joined #openstack-infra | 07:04 | |
| -openstackstatus- NOTICE: pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved | 07:06 | |
| *** ChanServ changes topic to "pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved" | 07:06 | |
| *** esikachev has joined #openstack-infra | 07:08 | |
| *** hparekh has joined #openstack-infra | 07:08 | |
| openstackstatus | jhesketh: finished sending alert | 07:08 |
| *** rcernin has quit IRC | 07:09 | |
| *** acruz has joined #openstack-infra | 07:10 | |
| *** rcernin has joined #openstack-infra | 07:10 | |
| ianw | jhesketh: yeah ... i think the thing is that do we have one canonical place to pin pip? | 07:10 |
| jhesketh | ianw: currently trying to figure that out... I think we might need to do it as a dib element | 07:10 |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 07:10 |
| *** e0ne has joined #openstack-infra | 07:11 | |
| ianw | jhesketh: system-config:install_puppet.sh is the first place | 07:12 |
| ianw | but ISTR last time pip broke, there were a couple of other places that might overwrite it. devstack for example | 07:12 |
| ianw | jhesketh: the other thing that might be done quickly is an apache rewrite rule | 07:13 |
| jhesketh | hmm interesting thought | 07:14 |
| *** markus_z has joined #openstack-infra | 07:14 | |
| jhesketh | pip will be installed on test nodes from dib, not install_puppet afaict... as you mention though devstack may do something else with it again | 07:14 |
| ianw | jhesketh: puppet dib calls that script, unless something has changed | 07:15 |
| *** thorst___ has quit IRC | 07:15 | |
| AJaeger | change Id12f866f577c3ca2405a6049084c3cb0af82fde5 was what we did for system-config last time | 07:16 |
| AJaeger | Not sure what else was changed ;( | 07:16 |
| *** thorst_ has joined #openstack-infra | 07:16 | |
| ianw | jhesketh: i'm thinking the redirect option might work, i *think* matching everything after /simple/ and s/-/./ might work | 07:17 |
| ianw | ohhh, unless it is supposed to have a '-' in it | 07:17 |
| *** yuanying has quit IRC | 07:18 | |
| *** amotoki has quit IRC | 07:18 | |
| ianw | so maybe could limit it to s/oslo-*/oslo./ ... urgh | 07:19 |
| jhesketh | ianw: It looks like puppet dib is only on centos-7 and fedora-23 unless it's pulled in elsewhere or I'm missing something... the pip-and-virtualenv element is pulled in by simple-init so I think that's where most nodes will have it from | 07:19 |
| jhesketh | ianw: we could have a condition on the apache rule to only be if the requested filename doesn't work | 07:20 |
| jhesketh | but if it has a mixture of .'s and -'s it'll be difficult | 07:20 |
| *** dimtruck is now known as zz_dimtruck | 07:21 | |
| ianw | hmm, yeah "if ! directory" might be good enough ... maybe the corner cases don't hit us | 07:21 |
| jhesketh | AJaeger: that change possibly still applies, but I'm not sure if it'll be ran on the test nodes | 07:21 |
| *** zeih has joined #openstack-infra | 07:22 | |
| ianw | $ less simple | grep '-' | grep '\.' | wc -l | 07:22 |
| ianw | 564 | 07:22 |
| ianw | that's packages with a "-" and a "." | 07:22 |
| ianw | out of 80339 | 07:23 |
| *** thorst_ has quit IRC | 07:24 | |
| *** vincentll has joined #openstack-infra | 07:24 | |
| *** amotoki has joined #openstack-infra | 07:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:26 | |
| *** daemontool_ has joined #openstack-infra | 07:26 | |
| *** gabriel-bezerra has joined #openstack-infra | 07:27 | |
| *** nmagnezi has quit IRC | 07:30 | |
| *** markvoelker_ has quit IRC | 07:30 | |
| *** yaume has joined #openstack-infra | 07:34 | |
| *** nmagnezi has joined #openstack-infra | 07:34 | |
| *** yolanda has joined #openstack-infra | 07:37 | |
| *** ifarkas has joined #openstack-infra | 07:37 | |
| *** hferenc has joined #openstack-infra | 07:37 | |
| *** jiggalator has quit IRC | 07:37 | |
| hferenc | hi all! can i bother you with a question: http://lists.openstack.org/pipermail/openstack-infra/2016-May/004215.html ? | 07:38 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503 https://review.openstack.org/314898 | 07:39 |
| jhesketh | ianw, AJaeger: something like ^ (note untested) | 07:39 |
| *** netsin has joined #openstack-infra | 07:39 | |
| *** jlanoux has joined #openstack-infra | 07:40 | |
| AJaeger | jhesketh: sorry, can't get my head around that. I love the idea but can't check the conditions ;( | 07:41 |
| jhesketh | I think I have the logic backwards actually... we should replace -'s with .'s from the request URL as it's the -'s that are requested and teh .'s that exist | 07:42 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503 https://review.openstack.org/314898 | 07:43 |
| jhesketh | AJaeger: this seems to be a useful tool http://htaccess.mwl.be/ | 07:43 |
| jhesketh | using http://mirror.ord.rax.openstack.org/pypi/simple/oslo-config/ the new url becomes http://mirror.ord.rax.openstack.org/pypi/simple/oslo.config/ | 07:44 |
| openstackgerrit | Ryo Tagami proposed openstack-infra/jenkins-job-builder: Fix Multiple SCMs Plugin child XML format https://review.openstack.org/314900 | 07:45 |
| *** kzaitsev_mb has quit IRC | 07:46 | |
| AJaeger | jhesketh: so, that looks fine ;) | 07:48 |
| jhesketh | I'm actually thinking it should be a passthrough... | 07:48 |
| * jhesketh reads up on mod_rewrite some more | 07:48 | |
| *** chlong has quit IRC | 07:49 | |
| *** claudiub has quit IRC | 07:50 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503 https://review.openstack.org/314898 | 07:51 |
| jhesketh | ianw: if you have a second to sanity check ^ | 07:51 |
| jhesketh | otherwise I'll apply it to a mirror manually and see how it goes | 07:51 |
| *** amotoki has quit IRC | 07:54 | |
| *** amotoki has joined #openstack-infra | 07:55 | |
| *** hashar has joined #openstack-infra | 07:58 | |
| *** oanson has quit IRC | 07:58 | |
| openstackgerrit | Sergey Gotliv proposed openstack-infra/project-config: Enable translation for manila-ui https://review.openstack.org/314905 | 07:59 |
| senk | someone able to +2 this one? https://review.openstack.org/#/c/310823/ | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** binbincong has quit IRC | 08:01 | |
| *** jordanP has joined #openstack-infra | 08:01 | |
| *** binbincong has joined #openstack-infra | 08:02 | |
| openstackgerrit | Yushiro FURUKAWA proposed openstack/requirements: Add networking-fujitsu into projects.txt https://review.openstack.org/314906 | 08:02 |
| *** vincentll has quit IRC | 08:03 | |
| *** claudiub has joined #openstack-infra | 08:03 | |
| *** markus_z has quit IRC | 08:03 | |
| *** amotoki has quit IRC | 08:04 | |
| *** ifarkas has quit IRC | 08:04 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-dev/cookiecutter: Don't include openstack/commmon in flake8 exclude list https://review.openstack.org/314907 | 08:05 |
| *** weshay has joined #openstack-infra | 08:07 | |
| *** dmk0202 has joined #openstack-infra | 08:08 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quotas support https://review.openstack.org/285110 | 08:08 |
| *** pavel_bondar has quit IRC | 08:10 | |
| *** pavel_bondar has joined #openstack-infra | 08:11 | |
| *** oanson has joined #openstack-infra | 08:11 | |
| *** zz_dimtruck is now known as dimtruck | 08:12 | |
| craige | There be dragons in mod_rewrite jordanP | 08:14 |
| craige | ^^ jhesketh | 08:14 |
| *** asettle has joined #openstack-infra | 08:14 | |
| *** fawadkhaliq has quit IRC | 08:14 | |
| *** fawadkhaliq has joined #openstack-infra | 08:15 | |
| ianw | jhesketh: LGTM, but i'm just going by reading the man page :) i feel like that's the simplest place to start | 08:17 |
| *** vincentll has joined #openstack-infra | 08:17 | |
| jhesketh | craige: as in you found something with my rules? | 08:17 |
| jhesketh | ianw: so I just tried applying it and it appeared to have no effect... trying to figure out why | 08:18 |
| ianw | urggh, mod_rewrite debugging is the worst | 08:18 |
| *** markus_z has joined #openstack-infra | 08:18 | |
| *** daemontool_ has quit IRC | 08:18 | |
| *** salv-orl_ has joined #openstack-infra | 08:19 | |
| *** ifarkas has joined #openstack-infra | 08:19 | |
| jhesketh | going to try with LogLevel warn rewrite:debug turned on | 08:20 |
| *** daemontool_ has joined #openstack-infra | 08:20 | |
| *** amotoki has joined #openstack-infra | 08:21 | |
| *** javeriak has quit IRC | 08:21 | |
| *** haaj has joined #openstack-infra | 08:22 | |
| *** amotoki has quit IRC | 08:22 | |
| *** vgridnev has quit IRC | 08:22 | |
| jhesketh | sigh, I need to restart apache to up the log levels... that might make a few jobs fail | 08:23 |
| *** vgridnev has joined #openstack-infra | 08:24 | |
| jhesketh | probably better than the amount of jobs about to fail though | 08:24 |
| *** fawadkhaliq has quit IRC | 08:25 | |
| *** fawadkhaliq has joined #openstack-infra | 08:25 | |
| *** amotoki has joined #openstack-infra | 08:26 | |
| *** dingyichen has quit IRC | 08:26 | |
| *** haaj has quit IRC | 08:29 | |
| *** yamahata has quit IRC | 08:29 | |
| *** Hal has joined #openstack-infra | 08:30 | |
| *** Hal is now known as Guest40739 | 08:30 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quotas support https://review.openstack.org/285110 | 08:30 |
| *** dimtruck is now known as zz_dimtruck | 08:31 | |
| *** haaj has joined #openstack-infra | 08:31 | |
| *** patrick_k has joined #openstack-infra | 08:34 | |
| *** yuanying has joined #openstack-infra | 08:34 | |
| patrick_k | hi, I want to add new experimental job for nova, how can I proceed? | 08:34 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 08:35 |
| jhesketh | ianw: okay, now I'm confused... http://paste.openstack.org/show/496664/ shows the rewrite rule working but I'm still getting 404 | 08:35 |
| rcarrillocruz | patrick_k: look for your project in project-config repo, zuul/layout.yaml | 08:35 |
| rcarrillocruz | add the job in the experimental stanza | 08:35 |
| *** Na3iL has joined #openstack-infra | 08:35 | |
| jhesketh | and /var/www/mirror/pypi/simple/o/oslo.config/index.html exists fyi | 08:35 |
| patrick_k | racrrillocruz: ok | 08:36 |
| patrick_k | rcarrillocruz: is there any reference patch for the same? | 08:37 |
| rcarrillocruz | patch reference? | 08:37 |
| patrick_k | like any experimental job example? | 08:37 |
| *** yuanying has quit IRC | 08:37 | |
| ianw | jhesketh: wait, it's "." to "-" ... unless i'm getting myself confused | 08:38 |
| *** jistr has joined #openstack-infra | 08:38 | |
| jhesketh | ianw: that's what I thought at first... but the request comes in as '-' from the new pip, and bandersnatch still mirrors with '.'s | 08:38 |
| jhesketh | so we're converting - to .'s from the mirrors point of view | 08:38 |
| rcarrillocruz | ah | 08:38 |
| *** yuanying has joined #openstack-infra | 08:38 | |
| rcarrillocruz | well, you could check on git log story of project-config | 08:39 |
| rcarrillocruz | also | 08:39 |
| rcarrillocruz | http://docs.openstack.org/infra/manual/creators.html | 08:39 |
| ianw | jhesketh: yeah, ok, pip has canonicalised it | 08:39 |
| *** vgridnev has quit IRC | 08:39 | |
| *** sambetts|afk is now known as sambetts | 08:39 | |
| patrick_k | rcarrillocruz: thank you, I will have a look | 08:40 |
| rcarrillocruz | np | 08:40 |
| ianw | jhesketh: ahh, hang on, it's rewriting the URL, right? there's something else for file level IIRC | 08:41 |
| jhesketh | ianw: oh? that might be more useful | 08:41 |
| *** yuanying has quit IRC | 08:41 | |
| ianw | or maybe actually it's the PT in that rule that's doing it | 08:42 |
| *** shardy has joined #openstack-infra | 08:43 | |
| *** yuanying has joined #openstack-infra | 08:43 | |
| *** rossella_s has joined #openstack-infra | 08:44 | |
| *** yuanying has quit IRC | 08:47 | |
| openstackgerrit | Sylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project https://review.openstack.org/312487 | 08:47 |
| openstackgerrit | Sylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project https://review.openstack.org/312487 | 08:48 |
| *** javeriak has joined #openstack-infra | 08:48 | |
| *** markvoelker has joined #openstack-infra | 08:48 | |
| *** amotoki has quit IRC | 08:49 | |
| *** amotoki has joined #openstack-infra | 08:50 | |
| frickler | jhesketh: ianw: what about names like "django_compressor"? IIUC the new pip would normalize that to "django-compressor", too? | 08:50 |
| *** derekh has joined #openstack-infra | 08:51 | |
| ianw | underscore too, yeah | 08:51 |
| jhesketh | yeah, just did a test and it looks like it :-( | 08:51 |
| openstackgerrit | Yaroslav Lobankov proposed openstack-infra/project-config: [Rally] Adding two new non-voiting jobs for `rally verify` https://review.openstack.org/310151 | 08:52 |
| *** haaj has quit IRC | 08:52 | |
| *** yuanying has joined #openstack-infra | 08:52 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 08:53 |
| *** vincentll has quit IRC | 08:54 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add planet blog aggregator feeds to website https://review.openstack.org/311910 | 08:54 |
| *** markvoelker has quit IRC | 08:55 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add Emilien Macchi blog to TripleO blogs feed https://review.openstack.org/313566 | 08:55 |
| jhesketh | frickler, ianw: looks like the old pkg safe_name already replaced underscores and bandersnatch handles those fine | 08:56 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add Emilien Macchi blog to TripleO blogs feed https://review.openstack.org/313566 | 08:57 |
| *** yuanying has quit IRC | 08:57 | |
| *** vkramskikh_ has joined #openstack-infra | 08:57 | |
| ianw | jhesketh: plain "RewriteRule /simple/(.*)-(.*) /simple/$1.$2" seems to work in local testing | 08:57 |
| jhesketh | ianw: but we actually need to rewrite into pypi/simple/a/alpha | 08:59 |
| jhesketh | which I guess we could do as multiple rewrite rules rather than a passthrough (which is what I had first) | 08:59 |
| *** HeOS has joined #openstack-infra | 09:02 | |
| jhesketh | ianw: thanks! I think that works now :-)... | 09:04 |
| jhesketh | no idea what was wrong with my passthrough rules | 09:04 |
| ianw | cool | 09:05 |
| jhesketh | ianw: http://mirror.ord.rax.openstack.org/pypi/simple/oslo-config/ | 09:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503 https://review.openstack.org/314898 | 09:05 |
| jhesketh | ianw: with ^ applied | 09:05 |
| *** markus_z has quit IRC | 09:05 | |
| jhesketh | err, without the extra logging | 09:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503 https://review.openstack.org/314898 | 09:05 |
| *** nijaba has quit IRC | 09:06 | |
| ianw | jhesketh: nice, pip 8.0.2 installs oslo.config from the mirror now | 09:06 |
| *** electrofelix has joined #openstack-infra | 09:06 | |
| openstackgerrit | Dmytro Dovbii proposed openstack-infra/project-config: Make gate-murano-python34-db not running on stable/mitaka https://review.openstack.org/314645 | 09:06 |
| *** vincentll has joined #openstack-infra | 09:07 | |
| *** andreas_s has quit IRC | 09:07 | |
| jhesketh | AJaeger: do you want to review 314898 before I merge | 09:08 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add non-voting docs job for fuel-web https://review.openstack.org/314920 | 09:08 |
| jhesketh | yolanda, SergeyLukjanov or if either of you are around ^ | 09:09 |
| *** jang has joined #openstack-infra | 09:10 | |
| penguinolog | https://review.openstack.org/#/c/314518/ - merge failed because of curretn topic (pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved)... | 09:10 |
| penguinolog | What should I do? | 09:10 |
| *** javeriak has quit IRC | 09:11 | |
| jhesketh | penguinolog: hold tight... we're working on a fix and will update the topic when finished | 09:11 |
| yolanda | jhesketh, i'm not available today sorry... still at the hospital | 09:11 |
| jhesketh | yolanda: no worries. Sorry to interrupt | 09:11 |
| penguinolog | jhesketh, no, my question is different: am I correct, that after repair I should rebase the patch? | 09:12 |
| *** nijaba has joined #openstack-infra | 09:13 | |
| jhesketh | penguinolog: no need to rebase... after it's fixed a recheck should do it | 09:13 |
| ianw | penguinolog: just a "recheck" will work | 09:13 |
| *** hashar has quit IRC | 09:13 | |
| jhesketh | okay lacking any other infra-root, I'm going to merge 314898 | 09:14 |
| *** hashar has joined #openstack-infra | 09:14 | |
| ianw | jhesketh: probably CI on https://review.openstack.org/#/c/314645/ will be a good test. should be a precise nodes on rax we run project-config tests on | 09:16 |
| *** markvoelker has joined #openstack-infra | 09:16 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-zuul: enable customize layout file name in zuul.conf https://review.openstack.org/314921 | 09:17 |
| jhesketh | ianw: yep | 09:17 |
| ianw | jhesketh: ahh, ran in IAD | 09:18 |
| ianw | https://review.openstack.org/#/c/313631/ might be | 09:19 |
| ianw | quite annoying to not have the live jenkins logs ATM | 09:19 |
| jhesketh | I'm not sure my hot fix will stayed applied anyway... since I think puppet will reload apache when it notices the vhost is wrong | 09:19 |
| openstackgerrit | Merged openstack-infra/system-config: Work around bandersnatch not supporting pep503 https://review.openstack.org/314898 | 09:19 |
| *** markus_z has joined #openstack-infra | 09:19 | |
| jhesketh | so we need to wait for the change to apply | 09:19 |
| jhesketh | (the alternative to disable puppet on each mirror and apply manually) | 09:19 |
| jhesketh | (but puppet runs are quick, so it can wait a few more minutes) | 09:20 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add non-voting docs job for fuel-web https://review.openstack.org/314920 | 09:20 |
| penguinolog | jhesketh, thank you! Just previously gates was not started, if previously failed by some error and recheck was used. | 09:20 |
| *** markvoelker has quit IRC | 09:20 | |
| *** tesseract has joined #openstack-infra | 09:20 | |
| *** daemontool_ has quit IRC | 09:21 | |
| *** andreas_s has joined #openstack-infra | 09:21 | |
| AJaeger | jhesketh: a recheck will only help once the issue is fixed. And rebase is never needed, since rebase is done as part of the job run | 09:21 |
| AJaeger | penguinolog: wanted to address you ^ | 09:22 |
| AJaeger | sorry, jhesketh h | 09:22 |
| *** daemontool_ has joined #openstack-infra | 09:22 | |
| jhesketh | :-) | 09:22 |
| ianw | jhesketh: yeah, that also ran in IAD. we'll soon find out :) | 09:22 |
| *** zz_dimtruck is now known as dimtruck | 09:23 | |
| *** thorst_ has joined #openstack-infra | 09:26 | |
| *** zeih has quit IRC | 09:28 | |
| *** sree__ has joined #openstack-infra | 09:28 | |
| skraynev | jhesketh: hi. could you please say update about current issue with oslo- ? I have seen your patch. does it enough for fix? | 09:29 |
| jhesketh | skraynev: it worked when manually tested... just waiting for it to apply | 09:30 |
| jhesketh | about to run the puppet manually | 09:30 |
| *** dtantsur|afk is now known as dtantsur | 09:30 | |
| jhesketh | ah, it's already running | 09:31 |
| *** sree has quit IRC | 09:32 | |
| *** fawadkhaliq has quit IRC | 09:32 | |
| *** fawadkhaliq has joined #openstack-infra | 09:32 | |
| *** dimtruck is now known as zz_dimtruck | 09:33 | |
| *** javeriak has joined #openstack-infra | 09:34 | |
| *** thorst_ has quit IRC | 09:35 | |
| *** amotoki has quit IRC | 09:35 | |
| skraynev | jhesketh: ok. let's wait a merge :) | 09:36 |
| jhesketh | it's merged... just waiting for puppet to finish | 09:36 |
| jhesketh | looks to be applied successfully http://mirror.regionone.bluebox-sjc1.openstack.org/pypi/simple/oslo-messaging/ | 09:38 |
| AJaeger | woot! | 09:39 |
| *** daemontool_ has quit IRC | 09:40 | |
| rakhmerov | are we good now? Can recheck our jobs? | 09:40 |
| AJaeger | rakhmerov: we're testing right now... | 09:40 |
| rakhmerov | ok | 09:41 |
| AJaeger | jhesketh: http://logs.openstack.org/20/314920/2/check/gate-project-config-layout/f8fc5de/console.html worked! | 09:43 |
| *** hparekh has quit IRC | 09:44 | |
| AJaeger | rakhmerov: feel free to recheck one or two changes and tell us whether it works, please | 09:44 |
| vponomaryov | it works | 09:44 |
| vponomaryov | http://logs.openstack.org/12/314712/4/check/gate-manila-python34-db/5db2c93/testr_results.html.gz | 09:44 |
| jhesketh | AJaeger: yay | 09:44 |
| rakhmerov | AJaeger: ok, thanks a lot guys! | 09:44 |
| jhesketh | I'm also rechecking a few things before removing the notice | 09:44 |
| AJaeger | jhesketh: I noticed - and did the same ;) | 09:45 |
| *** yaume has quit IRC | 09:47 | |
| *** nadya has joined #openstack-infra | 09:47 | |
| *** pcrews has quit IRC | 09:47 | |
| *** tosky has joined #openstack-infra | 09:47 | |
| *** zeih has joined #openstack-infra | 09:50 | |
| *** mescanef_ has joined #openstack-infra | 09:50 | |
| *** mescanef has quit IRC | 09:52 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories https://review.openstack.org/312666 | 09:52 |
| AJaeger | jhesketh: http://logs.openstack.org/35/313935/4/check/gate-project-config-irc-access/f3e9ebf/console.html | 09:52 |
| AJaeger | "No matching distribution found for backports.functools-lru-cache>=1.0.3" | 09:52 |
| *** camunoz has quit IRC | 09:52 | |
| *** camunoz has joined #openstack-infra | 09:53 | |
| *** yamamot__ has quit IRC | 09:54 | |
| *** sree__ has quit IRC | 09:54 | |
| AJaeger | and "No matching distribution found for openstack.nose-plugin>=0.7" | 09:54 |
| AJaeger | Log: http://logs.openstack.org/35/314435/3/check/gate-horizon-python34/1bd458a/console.html | 09:54 |
| *** zeih has quit IRC | 09:54 | |
| rakhmerov | AJaeger: I recheck one of our things and it's working fine (https://review.openstack.org/#/c/313257/) | 09:55 |
| AJaeger | So, on project-config the irc-access job is still failing - and on other repos there are still failures. | 09:55 |
| jhesketh | hmm | 09:55 |
| *** yamamot__ has joined #openstack-infra | 09:55 | |
| rakhmerov | rechecked | 09:55 |
| *** hparekh has joined #openstack-infra | 09:56 | |
| AJaeger | jhesketh: is the combination of "." and "-" a problem? | 09:56 |
| ianw | hmm, the rewritecond should just pass it through if there's a dir though? | 09:57 |
| jhesketh | well we thought that might be an edge case | 09:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting docs job for fuel-web https://review.openstack.org/314920 | 09:57 |
| apuimedo | a | 09:57 |
| AJaeger | http://mirror.regionone.bluebox-sjc1.openstack.org/pypi/simple/openstack.nose-plugin/ exists | 09:57 |
| openstackgerrit | Merged openstack-infra/project-config: fuel-devops: Add jobs https://review.openstack.org/314518 | 09:58 |
| apuimedo | AJaeger: from when https://review.openstack.org/#/c/313981/ is merged, how long can it take to have the repos created in gerrit so we can start pushing the changes? | 09:58 |
| AJaeger | apuimedo: it needs a puppet run - so rule of thumb: Wait one hour after merge | 09:58 |
| ianw | oh, yeah, we've lost the information | 09:59 |
| ianw | pip is asking for http://mirror.ord.rax.openstack.org/pypi/simple/openstack-nose-plugin/ | 09:59 |
| apuimedo | AJaeger: do you know who I can beg for a second r+2? | 10:00 |
| AJaeger | apuimedo: right now nobody ;) We're firefighting - see topic | 10:01 |
| AJaeger | apuimedo: somebody will review later the week for sure... | 10:01 |
| *** DevBox has quit IRC | 10:01 | |
| AJaeger | (for sure -> I hope) | 10:01 |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-murano-python34-db not running on stable/mitaka https://review.openstack.org/314645 | 10:02 |
| ianw | [iwienand@fedora19 tmp]$ less simple | grep 'openstack\.' | 10:02 |
| ianw | <a href='openstack.nose-plugin'>openstack.nose_plugin</a><br/> | 10:02 |
| ianw | openstack.nose-plugin is literally the only one with a "." and "-" in it ... i wonder if it's worth just a hard-code in the regex | 10:02 |
| ianw | (in the openstack namespace...) | 10:03 |
| ianw | backports.* has a few more | 10:03 |
| apuimedo | ouch | 10:03 |
| AJaeger | There's backports.functools-lru-cache as well | 10:04 |
| apuimedo | AJaeger: good luck with the pypi mirror | 10:04 |
| AJaeger | if it's only the two of them - hardcode ;) | 10:04 |
| *** zhurong has quit IRC | 10:04 | |
| AJaeger | thanks, apuimedo | 10:04 |
| jhesketh | I'm confused why it's not actually working though... | 10:04 |
| *** sdague has joined #openstack-infra | 10:05 | |
| *** sdague has quit IRC | 10:05 | |
| jhesketh | oh, I think I see | 10:05 |
| ianw | yeah, i feel like this will have to be solved quickly tomorrow US time by pypa, comments building in https://github.com/pypa/pip/issues/3666 | 10:05 |
| *** sdague has joined #openstack-infra | 10:05 | |
| jhesketh | >>> canonicalize_name('openstack.something-else') | 10:05 |
| jhesketh | 'openstack-something-else' | 10:05 |
| ianw | so quick hard-codes might be best | 10:05 |
| jhesketh | so that's the name pip will request... which means it's impossible to tell where periods or dashes should be | 10:05 |
| ianw | jhesketh: yep; by the time it's got to apache we've lost the data | 10:06 |
| ianw | matches for openstack.nose-plugin (seems to be an outlier) and backports.* might be enough | 10:07 |
| openstackgerrit | Daniel Berrange proposed openstack/requirements: Require oslo.concurrency >= 3.8.0 https://review.openstack.org/314946 | 10:07 |
| jhesketh | ianw: possibly... alternatively we could try replacing one dash at a time | 10:08 |
| jhesketh | working left to right | 10:08 |
| jhesketh | (this is all very hacky) | 10:08 |
| *** Qiming has quit IRC | 10:09 | |
| *** signed8bit has joined #openstack-infra | 10:09 | |
| *** gildub has quit IRC | 10:09 | |
| *** yaume has joined #openstack-infra | 10:09 | |
| *** markvoelker has joined #openstack-infra | 10:10 | |
| *** zeih has joined #openstack-infra | 10:12 | |
| *** dteselkin has quit IRC | 10:13 | |
| *** stevemar has quit IRC | 10:13 | |
| *** asilenkov has quit IRC | 10:13 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet/proposal: add missing publishers https://review.openstack.org/314751 | 10:13 |
| *** markvoelker has quit IRC | 10:15 | |
| *** stevemar has joined #openstack-infra | 10:15 | |
| *** andreas_s has quit IRC | 10:16 | |
| *** asilenkov has joined #openstack-infra | 10:16 | |
| *** daemontool has joined #openstack-infra | 10:18 | |
| *** dteselkin has joined #openstack-infra | 10:18 | |
| ianw | jhesketh: i think you'd have to try all permutations? | 10:19 |
| jhesketh | ianw: to be correct, yes, but to match the common pattern of 'namespace.package-name', less so | 10:19 |
| jhesketh | so long as namespace never has a dash in it | 10:20 |
| jhesketh | doing one at a time only really works with a passthrough though and we didn't get that working | 10:21 |
| *** claudiub has quit IRC | 10:21 | |
| jhesketh | so maybe special casing is the idea | 10:21 |
| ianw | sloth-ci.ext.shields-io | 10:21 |
| ianw | you name it, pypi has it :) | 10:21 |
| jhesketh | heh | 10:21 |
| jhesketh | I'm now curious what exactly that package is... | 10:22 |
| jhesketh | Status build badges for Sloth CI apps, powered by http://shields.io. | 10:22 |
| *** haaj has joined #openstack-infra | 10:23 | |
| ianw | jhesketh: gotta head out sorry, not that i was doing much. | 10:24 |
| *** zz_dimtruck is now known as dimtruck | 10:24 | |
| jhesketh | ianw: no worries.. Your help was very valuable, thank you! | 10:25 |
| ianw | one other thought was to stop bandersnatch, write a quick script to run over the mirror directory and symlink normalised names | 10:25 |
| tonyb | I was about to suggest that | 10:25 |
| jhesketh | yeah I had considered that but didn't like the idea of stopping bandersnatch | 10:25 |
| ianw | probably would work, but has to keep the old names for old python, so "if normalized_name != regualar_name; ln normalized_name regular_name" i guess | 10:26 |
| tonyb | or run s script over g-r to gernerate a reverse rewrite rule so that .../openstack.nose.plpugin get rewritten to .../openstack.nose-plugin | 10:26 |
| tonyb | and then load that into apache | 10:27 |
| * tonyb is fully aware he's missed hours of scrollback and these may have been rejected for $reasons | 10:27 | |
| *** esikachev has quit IRC | 10:27 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow lists of boards and worklists to be paged https://review.openstack.org/309481 | 10:28 |
| jhesketh | tonyb: I think you'd just do everything not just g-r | 10:28 |
| tonyb | jhesketh: Hmm okay that's a much bigger rewrite config, but it could still be generated form the 'simple' index right? | 10:31 |
| *** andreas_s has joined #openstack-infra | 10:32 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Further fixes for bandersnatch https://review.openstack.org/314956 | 10:33 |
| jhesketh | ianw, tonyb: ^ for specific workaround | 10:33 |
| jhesketh | tonyb: yes I don't see why not | 10:33 |
| *** hdd has joined #openstack-infra | 10:34 | |
| tonyb | jhesketh: aww man you missed sloth-ci.ext.shields-io ;P | 10:35 |
| jhesketh | heh, I probably missed a bunch | 10:35 |
| tonyb | :) | 10:35 |
| *** hdd has quit IRC | 10:36 | |
| *** haaj has quit IRC | 10:37 | |
| *** amrith is now known as _amrith_ | 10:38 | |
| *** kmartin has quit IRC | 10:38 | |
| *** dimtruck is now known as zz_dimtruck | 10:39 | |
| AJaeger | jhesketh, tonyb: the change covers the two failures I saw so far - let's see whether that one was blocking more... | 10:39 |
| *** kushal has joined #openstack-infra | 10:39 | |
| tonyb | AJaeger: fair call. | 10:40 |
| AJaeger | tonyb: I would try this now as well - but if more issues show up, we might do one of the other suggestions - but that's more involved... | 10:41 |
| *** amitgandhinz has quit IRC | 10:42 | |
| *** markus_z has quit IRC | 10:43 | |
| *** vkramskikh_ has quit IRC | 10:43 | |
| *** jed56 has quit IRC | 10:44 | |
| AJaeger | jhesketh: since there's no other infra-root around, do you want to self-approve 314956? | 10:45 |
| jhesketh | okay if we're happy with this hack I'll test it manually then self-approve | 10:46 |
| jhesketh | but I do agree if there are more edge cases doing symlinks may be the way to go | 10:46 |
| *** claudiub has joined #openstack-infra | 10:46 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack https://review.openstack.org/309383 | 10:48 |
| *** daemontool has quit IRC | 10:48 | |
| AJaeger | WE have at least 12 changes in the gate queue - and they're all green. So, your first change seems to have covered a large set of packages... | 10:49 |
| *** ri0 has joined #openstack-infra | 10:50 | |
| *** zeih has quit IRC | 10:51 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Improve the worklist filtering functionality https://review.openstack.org/310287 | 10:52 |
| *** kushal has quit IRC | 10:52 | |
| *** yamamot__ has quit IRC | 10:53 | |
| *** yamamot__ has joined #openstack-infra | 10:53 | |
| *** nadya has quit IRC | 10:54 | |
| *** zeih has joined #openstack-infra | 10:54 | |
| *** esikachev has joined #openstack-infra | 10:54 | |
| *** ri0 has quit IRC | 10:55 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_ipa_tinyipa-smoke job https://review.openstack.org/314966 | 10:55 |
| *** ri0 has joined #openstack-infra | 10:59 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Further fixes for bandersnatch https://review.openstack.org/314956 | 10:59 |
| * jhesketh is struggling to get the rule to work :-s | 10:59 | |
| jhesketh | any idea what I'm missing ^ | 10:59 |
| *** jistr is now known as jistr|mtg | 11:00 | |
| *** ccamacho has quit IRC | 11:00 | |
| jhesketh | with that applied http://mirror.ord.rax.openstack.org/pypi/simple/backports-functools-lru-cache/ still 404's | 11:00 |
| *** winggundamth has joined #openstack-infra | 11:02 | |
| sdague | backports-functools-lru-cache => backports.functools-lru-cache is what you are going for? | 11:04 |
| *** markvoelker has joined #openstack-infra | 11:04 | |
| sdague | jhesketh: the apache access logs typically show the file it tried to read, right? | 11:04 |
| jhesketh | sdague: yes & no | 11:05 |
| *** zeih has quit IRC | 11:06 | |
| *** dprince has joined #openstack-infra | 11:06 | |
| jhesketh | it just shows the GET request, not the actual files it looks up | 11:06 |
| *** amotoki has joined #openstack-infra | 11:07 | |
| jhesketh | I can turn on more verbose logging on the rewrite module but that requires a restart of apache (rather than reload) | 11:07 |
| jhesketh | and shouldn't be on for long as it's very verbose | 11:07 |
| *** ccamacho has joined #openstack-infra | 11:08 | |
| Daviey | fungi: Hey. Would you be happy to push kilo-eol tag and delete the branch? Ta | 11:08 |
| *** amitgandhinz has joined #openstack-infra | 11:08 | |
| *** tellesnobrega is now known as tellesnobrega_af | 11:08 | |
| *** markvoelker has quit IRC | 11:09 | |
| jhesketh | ah, I figured it out... fix incoming | 11:10 |
| *** baoli has joined #openstack-infra | 11:10 | |
| sdague | jhesketh: RewriteRule (.*)-(.*) $1.$2 also seems weird, and is going to by default change the last - to . | 11:11 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Further fixes for bandersnatch https://review.openstack.org/314956 | 11:12 |
| *** Qiming has joined #openstack-infra | 11:12 | |
| *** baoli_ has joined #openstack-infra | 11:12 | |
| jhesketh | sdague: yeah, that's the workaround we currently have but there are some edge cases that ^ is going to address... if there are more edge cases we may undo that and symlink the new canonicalized names | 11:12 |
| jhesketh | AJaeger: ^ 314956 applies and tests okay now FYI | 11:12 |
| jhesketh | sdague: either way these are just workarounds until bandersnatch is fixed | 11:13 |
| sdague | jhesketh: the [N] is something I'm unfamiliar with | 11:13 |
| jhesketh | sdague: basically keep applying the rule until it can't any further | 11:14 |
| jhesketh | sdague: it's so the (.*) aren't greedy and only replace one - as a result | 11:14 |
| jhesketh | ie, replace all -'s with .'s | 11:14 |
| sdague | ok | 11:14 |
| *** jlanoux_ has joined #openstack-infra | 11:14 | |
| jhesketh | (as an aside I had initially tried to do this as a passthrough which would have allowed us to replace one dash at a time and avoid repeating the sub-directory rules but getting that to work was trickier than it was worth) | 11:15 |
| AJaeger | thanks, jhesketh | 11:15 |
| *** baoli has quit IRC | 11:16 | |
| *** rraja has joined #openstack-infra | 11:16 | |
| jhesketh | sdague: I was about to self-approve the extra workaround unless you had any thoughts/objections? | 11:17 |
| sdague | nope, seems as good as anything to me | 11:17 |
| jhesketh | ta | 11:18 |
| *** jlanoux has quit IRC | 11:18 | |
| *** mrunge has joined #openstack-infra | 11:20 | |
| *** rodrigods has quit IRC | 11:20 | |
| *** rodrigods has joined #openstack-infra | 11:20 | |
| sdague | jhesketh: we good to recheck? | 11:20 |
| jhesketh | sdague: yep, most jobs are passing, just these edge cases | 11:20 |
| jhesketh | I'll remove the notice once this merges and applies | 11:20 |
| * jhesketh lines up an email also | 11:21 | |
| *** nadya has joined #openstack-infra | 11:21 | |
| *** baoli_ has quit IRC | 11:21 | |
| *** baoli has joined #openstack-infra | 11:22 | |
| *** jaosorior has quit IRC | 11:22 | |
| *** jaosorior has joined #openstack-infra | 11:22 | |
| *** icey has joined #openstack-infra | 11:23 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add api-ref jobs for neutron https://review.openstack.org/314977 | 11:23 |
| openstackgerrit | Merged openstack-infra/system-config: Further fixes for bandersnatch https://review.openstack.org/314956 | 11:24 |
| *** baoli has quit IRC | 11:25 | |
| *** baoli has joined #openstack-infra | 11:25 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Fix validation for stable branch https://review.openstack.org/312350 | 11:26 |
| *** nadya has quit IRC | 11:26 | |
| dstufft | ianw: good catch re: bandersnatch patch | 11:27 |
| dstufft | I fixedorized it | 11:27 |
| *** tpsilva has joined #openstack-infra | 11:28 | |
| dstufft | (That's what I get for writing patches when I'm half asleep) | 11:28 |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** kushal has joined #openstack-infra | 11:33 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add api-ref jobs for neutron https://review.openstack.org/314977 | 11:34 |
| *** thorst_ has joined #openstack-infra | 11:37 | |
| ansiwen | AJaeger: can I see the semi-automatic changes resulting from the requirements change? | 11:37 |
| ansiwen | AJaeger: or better said: how can I find it? | 11:37 |
| *** rhallisey has joined #openstack-infra | 11:38 | |
| skraynev | AJaeger: hi. looks like I can not catch Anita Kuno to re-review my patch. could you say who else can help me and have rights for approve in project-config? | 11:39 |
| AJaeger | ansiwen: https://review.openstack.org/#/q/topic:openstack/requirements shows all reviews produced | 11:39 |
| rcarrillocruz | pabelanger: http://grafana.org/blog/2016/05/11/grafana-3-0-stable-released.html | 11:40 |
| ansiwen | AJaeger: thanks! | 11:40 |
| rcarrillocruz | (heya) | 11:40 |
| AJaeger | ansiwen: and you can check the post file from requirements like http://logs.openstack.org/24/24925ec82bc7ecc88cb37fd48864be4ee6b29c67/ | 11:40 |
| rcarrillocruz | just randomly saw it on twitter | 11:40 |
| AJaeger | skraynev: please have some patience - and do not rebase your changes! - see topic, we currently have to fix infra... | 11:40 |
| AJaeger | skraynev: reviews here typically take a week to merge... | 11:41 |
| *** salv-orl_ has quit IRC | 11:43 | |
| *** ccamacho is now known as ccamacho|lunch | 11:43 | |
| openstackgerrit | Sergey Gotliv proposed openstack-infra/project-config: Enable translation jobs for manila-ui https://review.openstack.org/314905 | 11:44 |
| *** thorst_ has quit IRC | 11:44 | |
| *** |-paul-| has joined #openstack-infra | 11:45 | |
| *** fawadkhaliq has quit IRC | 11:46 | |
| *** fawadkhaliq has joined #openstack-infra | 11:47 | |
| skraynev | AJaeger: Opps. My apologize . I have not know about such estimates. Then I will wait. thank you. I thought, that it happens quickly. | 11:48 |
| *** ociuhandu has joined #openstack-infra | 11:49 | |
| AJaeger | skraynev: sometimes it happens much faster but the one week is more realistic | 11:49 |
| AJaeger | jhesketh: irc-access job just passed, so one step forward ;) | 11:50 |
| *** kgiusti has joined #openstack-infra | 11:50 | |
| jhesketh | yep, just watched that :-) | 11:50 |
| jhesketh | just waiting for 314435 to go through | 11:50 |
| AJaeger | ok | 11:50 |
| *** _amrith_ is now known as amrith | 11:51 | |
| *** nadya has joined #openstack-infra | 11:51 | |
| *** hdd has joined #openstack-infra | 11:51 | |
| *** lucasagomes is now known as lucas-hungry | 11:52 | |
| AJaeger | horizon-python34 passed on https://review.openstack.org/#/c/309569/ - so, hope it passes on 314435 as well | 11:53 |
| jhesketh | yep, and pep8 pulled in openstack.nose-plugin 314435 | 11:54 |
| jhesketh | which is good enough for me :-) | 11:54 |
| jhesketh | #status ok We have a workaround for our mirrors to attempt to translate package names if a match isn't immediately obvious. A more complete fix is yet to come. It is now safe to 'recheck' any jobs that failed due to "No matching distribution found". Please join #openstack-infra if you discover more problems. | 11:55 |
| openstackstatus | jhesketh: sending ok | 11:55 |
| AJaeger | jhesketh: great, thanks! | 11:55 |
| jhesketh | AJaeger: Thank you for your help :-) | 11:56 |
| *** yamamot__ has quit IRC | 11:56 | |
| ansiwen | AJaeger: and how to proceed, if there is a verify failure: https://review.openstack.org/#/c/313743/ | 11:56 |
| sdague | jhesketh: hmmm.... Jenkins is no longer rolling up in hideci - https://review.openstack.org/#/c/314802/ | 11:56 |
| sdague | any thoughts as to why? | 11:56 |
| *** yamamot__ has joined #openstack-infra | 11:56 | |
| jhesketh | sdague: appear to for me | 11:57 |
| jhesketh | try clearing your cookies/cache? | 11:57 |
| jhesketh | or just refreshign the page | 11:57 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 11:57 | |
| -openstackstatus- NOTICE: We have a workaround for our mirrors to attempt to translate package names if a match isn't immediately obvious. A more complete fix is yet to come. It is now safe to 'recheck' any jobs that failed due to "No matching distribution found". Please join #openstack-infra if you discover more problems. | 11:57 | |
| sdague | jhesketh: Jenkins shows up? | 11:57 |
| dims | jhesketh : nice work with the rewrite rules! just catching up | 11:57 |
| *** psilvad has joined #openstack-infra | 11:58 | |
| jhesketh | sdague: oh sorry, I see.. I thought you meant in the comment (with the toggle CI).. no I don't see it in the summary | 11:58 |
| *** markvoelker has joined #openstack-infra | 11:58 | |
| sdague | https://dl.dropboxusercontent.com/1/view/zwtnsvi0rcj9m68/Apps/Shutter/screenshot_314.png | 11:58 |
| sdague | right, I see the others, not jenkins | 11:58 |
| jhesketh | sdague: ditto | 11:58 |
| sdague | and this has been true for a whole bunch of changes from last night | 11:59 |
| *** tosky has quit IRC | 11:59 | |
| ttx | Gerrit slowing down, might be the hundreds of people showing up to recheck changes | 12:00 |
| openstackstatus | jhesketh: finished sending ok | 12:00 |
| *** yamamot__ has quit IRC | 12:01 | |
| *** yamamot__ has joined #openstack-infra | 12:01 | |
| *** yamamot__ has quit IRC | 12:01 | |
| AJaeger | ansiwen: if there's a Jenkins -1, always check the log file first for the cause and figure out what it is. | 12:01 |
| *** salv-orlando has joined #openstack-infra | 12:01 | |
| AJaeger | ansiwen: if it was the "No matching distribution" found, add a recheck. Otherwise debug it ;) | 12:02 |
| *** yamamot__ has joined #openstack-infra | 12:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new project Higgins to openstack repo https://review.openstack.org/313935 | 12:02 |
| *** markvoelker has quit IRC | 12:03 | |
| *** zeih has joined #openstack-infra | 12:03 | |
| *** javeriak has quit IRC | 12:03 | |
| AJaeger | jhesketh: Looks great so far! | 12:03 |
| *** zeih has quit IRC | 12:07 | |
| *** claudiub has quit IRC | 12:08 | |
| jhesketh | sdague: the javascript is parsing the comments and building the data structures correctly... I'm not sure why it isn't rendering it | 12:08 |
| jhesketh | *well adding it to the dom | 12:08 |
| *** claudiub has joined #openstack-infra | 12:08 | |
| *** ddieterly has joined #openstack-infra | 12:10 | |
| jhesketh | sdague: oh I see... somehow Jenkins and Intel NFV have been rolled up into the same group... | 12:10 |
| jhesketh | sdague: that '1 rechecks' you see on the Intel CI is actually counting Jenkins, not 2 votes from intel | 12:11 |
| *** jistr|mtg is now known as jistr | 12:11 | |
| sdague | jhesketh: huh, any idea why? | 12:11 |
| jhesketh | still poking | 12:11 |
| sdague | yeh, I turned on the debug parsing (so glad I left that in there) | 12:11 |
| sdague | so I could see it's finding the entries | 12:11 |
| *** zeih has joined #openstack-infra | 12:11 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add ironic cleaning job https://review.openstack.org/304473 | 12:12 |
| sdague | jhesketh: so... notably, it's the only CI that we're getting a pipeline for | 12:13 |
| sdague | I wonder if there is a bad fold here | 12:13 |
| jhesketh | sdague: not sure what you mean by that? | 12:14 |
| openstackgerrit | Pranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting. https://review.openstack.org/314998 | 12:14 |
| jhesketh | sdague: oh, I see, on the parse comments | 12:14 |
| jhesketh | that is interesting... | 12:15 |
| sdague | https://dl.dropboxusercontent.com/1/view/obx9sdp9ql1zbj0/Apps/Shutter/screenshot_315.png | 12:15 |
| mordred | morning all | 12:15 |
| * mordred is in an all day meeting - but lurking here becuase here is more interesting | 12:16 | |
| *** zeih has quit IRC | 12:16 | |
| sdague | jhesketh: yeh, I think that's the issue | 12:16 |
| jhesketh | sdague: jenkins has the check pipeline too... they are also the only CI bots posting the pipeline in the format we expect | 12:17 |
| sdague | jhesketh: right | 12:17 |
| AJaeger | mordred: you missed the excitement - jhesketh has done apache rewrite voodoo to fix pip 8.1.2 ;) | 12:17 |
| *** patrick_k has quit IRC | 12:17 | |
| mordred | WOOT! | 12:17 |
| mordred | what did 8.1.2 do? | 12:17 |
| mordred | (also, nice) | 12:18 |
| AJaeger | http://lists.openstack.org/pipermail/openstack-dev/2016-May/094588.html | 12:18 |
| jhesketh | sdague: right, I think I understand the issue now... perhaps we need to group by ci_name + pipeline? | 12:18 |
| sdague | jhesketh: I think so | 12:18 |
| AJaeger | mordred: and see ianw and dstufft discussing at around 3:00 UTC | 12:18 |
| jhesketh | sdague: I can take a look at that if you like | 12:18 |
| *** amitgandhinz has quit IRC | 12:18 | |
| mordred | AJaeger: woot | 12:18 |
| sdague | jhesketh: that would be awesome | 12:18 |
| sdague | mordred: fwiw, it's not just 8.1.2 | 12:18 |
| sdague | I have an 8.1.1 system that sees similar thing | 12:19 |
| *** signed8bit is now known as signed8bit_Zzz | 12:19 | |
| sdague | pip search oslo.vmware | 12:19 |
| dstufft | pip search is kinda borken | 12:19 |
| dstufft | mostly because PyPI search is kind of borken | 12:19 |
| dstufft | an unrelated kind of borken | 12:19 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: change email topic tags based on summit feedback https://review.openstack.org/312762 | 12:20 |
| *** asettle has quit IRC | 12:20 | |
| AJaeger | jhesketh: if you fix this, please look also at https://review.openstack.org/308731 which was reverted - so that "wrong requirements" shows up in the list as well. Right now, they are filtered out | 12:20 |
| mordred | jianghuaw: I agree - what you describe is a bug and we should fix it | 12:20 |
| mordred | jhesketh: nice rewrite | 12:20 |
| *** hdd has quit IRC | 12:20 | |
| jhesketh | mordred: it's a total hack.. (my earlier version was slightly neater using passthroughs but turned out to be trickier than it was worth) | 12:21 |
| *** acruz has quit IRC | 12:21 | |
| *** ddieterly is now known as ddieterly[away] | 12:21 | |
| jhesketh | AJaeger: sure, I'll tackle the summary box first though | 12:22 |
| AJaeger | jhesketh: sure! One thing after the other ;) But first some sleep... | 12:22 |
| mmedvede | sdague: jhesketh : we can also ask wznoinsk to remove "(check pipeline)" from the Intel NFV CI comment | 12:22 |
| *** anteaya has joined #openstack-infra | 12:23 | |
| *** kashyap has left #openstack-infra | 12:23 | |
| sdague | mmedvede: we could, but honestly there is no reason why they should | 12:23 |
| sdague | they have a real zuul now | 12:23 |
| sdague | it's a bug in the parser | 12:24 |
| *** marcusvrn_ has joined #openstack-infra | 12:24 | |
| *** acruz has joined #openstack-infra | 12:24 | |
| mmedvede | sdague: it is a bug in the parser. Adding pipeline to the comment is not automatic, is it? | 12:25 |
| jhesketh | mmedvede: was about to reply exactly what sdague already said | 12:25 |
| jhesketh | mmedvede: zuul adds the pipeline to the comment.. so it's because they have configured zuul to have a pipeline called 'check' | 12:26 |
| jhesketh | if they called it 'my_ci_pipeline' it'd work | 12:26 |
| jhesketh | but as pointed out, it's a bug in the parser | 12:26 |
| *** ccamacho|lunch is now known as ccamacho | 12:28 | |
| mmedvede | ok, I wrongly assumed the pipeline was manually added | 12:28 |
| mordred | dstufft, jhesketh: is there a time in the future when bandersnatch will normalize things on the disk? | 12:29 |
| dstufft | mordred: there's a PR already | 12:29 |
| mordred | cool | 12:29 |
| dstufft | I talked to Christian, he's swamped with work stuff ATM but is going to try and merge it and get a release out as quickly as he can | 12:29 |
| mordred | so the hack is only needed for a little bit | 12:30 |
| *** akijak has joined #openstack-infra | 12:30 | |
| *** vincentll has quit IRC | 12:30 | |
| mordred | dstufft: while he's at it - it would be _great_ if he'd also land the hashed-directory patch we're running in production so we could start following his releases again | 12:30 |
| mordred | dstufft: but I know you don't have magic powers | 12:30 |
| *** zz_dimtruck is now known as dimtruck | 12:31 | |
| openstackgerrit | Graham Hayes proposed openstack/requirements: Block dnspython 1.13.0 https://review.openstack.org/315003 | 12:32 |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** openstackgerrit has joined #openstack-infra | 12:33 | |
| *** aysyd has joined #openstack-infra | 12:33 | |
| zigo | mordred: Hi there. Could you have a look at https://review.openstack.org/#/c/311190/ and let me know what is wrong? I'm not sure I understand pabelanger's comments. | 12:34 |
| openstackgerrit | Graham Hayes proposed openstack/requirements: Block dnspython 1.13.0 https://review.openstack.org/315003 | 12:34 |
| *** signed8bit_Zzz is now known as signed8bit | 12:34 | |
| mordred | zigo: lookig | 12:34 |
| *** thorst has joined #openstack-infra | 12:34 | |
| wznoinsk | mmedvede I feel check pipeline is a good name for our main pipeline and most likely other deployments of downstream-puppet will think the same | 12:35 |
| mmedvede | sdague: actually, the pipeline name in zuul result is manually configured, it is not added automatically. | 12:35 |
| mordred | zigo: jenkins will only upload files that are inside of its "workspace" - so you can't, in the publisher, get at things that are up a dir | 12:35 |
| zigo | mordred: How to work around this then? | 12:36 |
| mordred | zigo: you'll want to copy/move the output things that need to be uploaded into the workspace at the end of the build script | 12:36 |
| zigo | Ah. | 12:36 |
| wznoinsk | it's a zuul layout config (the pipeline name) and zuul layout pipeline's description that adds the '(check pipeline)' to the comment in gerrit | 12:36 |
| mordred | so that they're in a location you can reference here | 12:36 |
| zigo | That's messy ... :) | 12:36 |
| mmedvede | wznoinsk: the parser confused your CI with jenkins due to the comment having similar pattern, i.e. '(check pipeline)'. I remember you changed both your failure and success comment yesterday. I thought you would change it to remove the pipeline name. | 12:36 |
| *** ayoung has quit IRC | 12:36 | |
| mordred | zigo: yes. this will be better in zuul v3 - but for now it's what we've got :) | 12:36 |
| zigo | Thanks. | 12:36 |
| zigo | Doing this now. | 12:37 |
| mordred | zigo: woot! | 12:37 |
| *** Liuqing has joined #openstack-infra | 12:37 | |
| wznoinsk | mmedvede we also have different pipelines, and want to distinguish comments/results from different ones ... also 'check' is a good name as well | 12:37 |
| wznoinsk | mmedvede if it breaks things and can't be fixed in a short time on your side I can change my configuration (the comments only, the pipeline will remain 'check') for the time being | 12:38 |
| *** eharney has joined #openstack-infra | 12:39 | |
| *** thorst has quit IRC | 12:39 | |
| mmedvede | wznoinsk: as been said, it is still hideci parsing problem. Your CI just seemed to be the first to actually have the similar pattern to jenkins | 12:39 |
| *** yamamot__ has quit IRC | 12:40 | |
| *** yamamoto has joined #openstack-infra | 12:40 | |
| wznoinsk | mmedvede so you ok me leaving it like this ? | 12:40 |
| wznoinsk | btw. ci watch (http://ci-watch.tintri.com/project?project=nova) looks to be ok tome | 12:40 |
| *** dimtruck is now known as zz_dimtruck | 12:41 | |
| mmedvede | wznoinsk: consensus is the proper action is to fix hideci parser | 12:42 |
| *** mrkelley has joined #openstack-infra | 12:43 | |
| *** amitgandhinz has joined #openstack-infra | 12:44 | |
| mmedvede | wznoinsk: although ciwatch also uses the same pattern to distinguish jenkins from all other CI | 12:44 |
| *** ddieterly[away] is now known as ddieterly | 12:44 | |
| mmedvede | s | 12:44 |
| mmedvede | wznoinsk: so it needs to be fixed too | 12:44 |
| *** jamielennox is now known as jamielennox|away | 12:45 | |
| *** abregman is now known as abregman|afk | 12:45 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Allow 3rd party CI's to have same pipeline names https://review.openstack.org/315015 | 12:45 |
| jhesketh | sdague, mmedvede: ^ | 12:45 |
| *** vincentll has joined #openstack-infra | 12:45 | |
| *** amitgandhinz has quit IRC | 12:45 | |
| *** amitgandhinz has joined #openstack-infra | 12:46 | |
| pabelanger | zigo: jenkins will not access files in ../ only ./, so if you try to point jenkins to ../build-area, the publisher will fail. | 12:46 |
| zigo | pabelanger: Yup, I'm fixing deb-openstack-pkg-tools first. | 12:46 |
| *** asettle has joined #openstack-infra | 12:47 | |
| zigo | pabelanger: About your comment: lets use 'tarballs/{project}' to be more consitent with other projects on tarballs.o.o | 12:47 |
| zigo | pabelanger: Then thing is, that's supposed to be a temp folder which we would later on clean-up (every week?). | 12:47 |
| zigo | So I thought it made sense to have a known folder name. | 12:48 |
| zigo | Your thoughts? | 12:48 |
| sdague | jhesketh: that seems about as good as I can imagine it | 12:48 |
| zigo | Or is it that it's the case for every projects anyway? | 12:48 |
| *** sree has joined #openstack-infra | 12:48 | |
| pabelanger | rcarrillocruz: Ya, see that. Should see if grafyaml works still | 12:48 |
| *** lucas-hungry is now known as lucasagomes | 12:49 | |
| pabelanger | zigo: Ya, I'm fine with tarballs/{project} for now, we need to implement the clean-up stuff too | 12:50 |
| zigo | :) | 12:50 |
| *** daemontool has joined #openstack-infra | 12:51 | |
| cschwede | sdague: Hi! if you have a few minutes, could you have a look at https://review.openstack.org/#/c/304465/ and let me know what you think about it? that would be great! | 12:51 |
| *** links has quit IRC | 12:52 | |
| *** amitgandhinz has quit IRC | 12:52 | |
| *** markvoelker has joined #openstack-infra | 12:52 | |
| *** amitgandhinz has joined #openstack-infra | 12:52 | |
| *** xyang has joined #openstack-infra | 12:53 | |
| *** daemontool has quit IRC | 12:53 | |
| *** timl0101 has joined #openstack-infra | 12:53 | |
| *** jamielennox|away is now known as jamielennox | 12:53 | |
| *** daemontool has joined #openstack-infra | 12:53 | |
| *** bdemers has joined #openstack-infra | 12:54 | |
| *** smarcet has joined #openstack-infra | 12:54 | |
| *** psilvad has quit IRC | 12:55 | |
| *** sdake has joined #openstack-infra | 12:55 | |
| *** psilvad has joined #openstack-infra | 12:55 | |
| zigo | Was web access to jenkins<num>.openstack.org removed? | 12:56 |
| zigo | https://jenkins02.openstack.org/job/pkgdeb-build-openstack-pkg-tools/12/ <=== Canm' | 12:56 |
| zigo | can't get there | 12:56 |
| sdague | cschwede: cool, I've got it starred I'll circle around on it when I can dump state | 12:56 |
| *** regXboi has joined #openstack-infra | 12:56 | |
| cschwede | sdague: thx a lot! | 12:56 |
| *** rfolco has joined #openstack-infra | 12:57 | |
| *** markvoelker has quit IRC | 12:57 | |
| sdague | mordred / fungi - https://review.openstack.org/#/c/315015/ approving this would be extremely helpful for any project where Intel NFV CI is commenting, as right now it is typically masking out Jenkins votes | 12:57 |
| *** Goneri has joined #openstack-infra | 12:58 | |
| *** xyang1 has joined #openstack-infra | 12:58 | |
| *** sdake_ has joined #openstack-infra | 12:58 | |
| *** tlian has joined #openstack-infra | 12:59 | |
| pabelanger | zigo: yes, we firewall it off because of a security issue | 12:59 |
| pabelanger | so, no more public access to jenkins UI | 13:00 |
| *** aysyd has quit IRC | 13:00 | |
| zigo | I was expecting something like that. | 13:00 |
| *** psilvad has quit IRC | 13:00 | |
| zigo | Jenkins is a @#$(%&* | 13:00 |
| anteaya | jhesketh ianw sdague thanks for putting the workaround in place | 13:01 |
| *** xyang1 has quit IRC | 13:01 | |
| *** dtardivel has joined #openstack-infra | 13:01 | |
| *** ddieterly has quit IRC | 13:02 | |
| *** sdake has quit IRC | 13:02 | |
| *** zqfan has quit IRC | 13:03 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Publish debian packages after build https://review.openstack.org/311190 | 13:03 |
| daemontool | yolanda, ping at your convenience if we could get +A here: https://review.openstack.org/#/c/314285/ New repo for the Freezer Project | 13:03 |
| *** edmondsw has joined #openstack-infra | 13:04 | |
| *** acruz has quit IRC | 13:04 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Removed explicit references to node_modules/.bin https://review.openstack.org/304557 | 13:05 |
| *** thorst_ has joined #openstack-infra | 13:05 | |
| *** kengo_sakai has quit IRC | 13:06 | |
| *** amotoki has quit IRC | 13:08 | |
| *** mrkelley has left #openstack-infra | 13:08 | |
| *** Nakato has quit IRC | 13:08 | |
| *** Nakato has joined #openstack-infra | 13:08 | |
| *** rlandy has joined #openstack-infra | 13:08 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Make the reviewday config file pluggable https://review.openstack.org/307377 | 13:09 |
| *** matt-borland has joined #openstack-infra | 13:10 | |
| smarcet | fungi: morning, how are u doing? still having problems with tars generation on openstackid | 13:10 |
| *** andymaier has joined #openstack-infra | 13:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Delete bare-precise https://review.openstack.org/313631 | 13:12 |
| anteaya | I'm mostly offline today, running errands | 13:12 |
| pabelanger | anteaya: enjoy | 13:12 |
| anteaya | pabelanger: thanks | 13:12 |
| *** gordc has joined #openstack-infra | 13:13 | |
| anteaya | pabelanger: I talked with shamail about the wiki, he would like to help if he can be shown/told what to do, he said he would pop into channel today | 13:13 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Ignore bash errors when posting metrics https://review.openstack.org/307374 | 13:13 |
| anteaya | pabelanger: if he does while I'm out can you give him a bit of an orientation? | 13:13 |
| clarkb | smarcet: http://logs.openstack.org/a8/a842c9c0e732a06e6ed2bb23313eaf80f7ad2911/post/openstackid-release-branch/1755183/console.html#_2016-05-10_22_08_26_087 | 13:13 |
| clarkb | smarcet: looks like the mcrypt enablement didnt take? | 13:14 |
| *** jsavak has joined #openstack-infra | 13:14 | |
| *** aysyd has joined #openstack-infra | 13:14 | |
| clarkb | or rather fixed it in one test buy not the other | 13:14 |
| *** alikhya has joined #openstack-infra | 13:15 | |
| smarcet | clarkb: weird, this error just arose in these days | 13:15 |
| smarcet | on the past we didnt have that, should be related to the moving to trusty | 13:16 |
| pabelanger | anteaya: sure | 13:16 |
| *** mriedem has joined #openstack-infra | 13:16 | |
| *** pabelanger has quit IRC | 13:16 | |
| *** pabelanger has joined #openstack-infra | 13:16 | |
| anteaya | pabelanger: thanks, I appreciate it | 13:16 |
| ansiwen | AJaeger: but I don't get that, so my change is merged into requirements, which creates a lot of followup changes in all the projects. there are a couple of failures. while I'm looking at these, other people will merge more stuff into requirements... shouldn't "requirements" then be gated with all the project merges its producing? | 13:17 |
| *** alikhya has quit IRC | 13:18 | |
| *** amitgandhinz has quit IRC | 13:18 | |
| smarcet | clarkb: weird thing is that https://review.openstack.org/#/c/314403/ ran ok | 13:19 |
| smarcet | clarkb: not sure if there is any other job to fix | 13:19 |
| *** aysyd has quit IRC | 13:19 | |
| *** amitgandhinz has joined #openstack-infra | 13:20 | |
| clarkb | smarcet: the job that failed needs to be fixed | 13:20 |
| *** ffrank has joined #openstack-infra | 13:20 | |
| *** ri0 has quit IRC | 13:20 | |
| clarkb | openstackid-release-branch | 13:20 |
| smarcet | clarkb: ok sure, | 13:20 |
| smarcet | clarkb: its located on project config ? | 13:21 |
| clarkb | yes, should be in the same dir as the previous one | 13:21 |
| SotK | hey folks, is there a way I can find out what versions of things are in use on storyboard.openstack.org? | 13:21 |
| smarcet | ok thx | 13:21 |
| SotK | specifically oslo_db, sqlalchemy, and sqlalchemy-fulltext-search | 13:21 |
| mordred | clarkb: I just sent up two patches to swift adding godep infrastructure: https://review.openstack.org/315028 and https://review.openstack.org/315029 | 13:22 |
| mordred | sdague: lgtm - but I'd prefer someone else look at it too | 13:25 |
| openstackgerrit | sebastian marcet proposed openstack-infra/project-config: Fix on OpenstackID job https://review.openstack.org/315031 | 13:27 |
| *** arxcruz has joined #openstack-infra | 13:28 | |
| *** ddieterly has joined #openstack-infra | 13:28 | |
| *** vincentll has quit IRC | 13:28 | |
| *** hamzy has quit IRC | 13:29 | |
| smarcet | clarkb: https://review.openstack.org/#/c/315031/ done :) | 13:29 |
| *** _ari_ has joined #openstack-infra | 13:29 | |
| *** hamzy has joined #openstack-infra | 13:29 | |
| akijak | hi, I'd like to create a new channel: #openstack-pl (general disscussions, support in Polish). I've read https://wiki.openstack.org/wiki/IRC and here I am :) Can you help? | 13:29 |
| *** sree__ has joined #openstack-infra | 13:29 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add networking-midonet grafana dashboard https://review.openstack.org/315033 | 13:31 |
| fungi | SotK: SQLAlchemy==0.9.8, SQLAlchemy-FullText-Search==0.2, sqlalchemy-migrate==0.9.4 | 13:31 |
| SotK | fungi: thanks! | 13:32 |
| *** sree has quit IRC | 13:32 | |
| clarkb | smarcet: approved, once that gets applied someone will need to retrigger that job or wait for new merge to happen for new tarball | 13:33 |
| *** eharney has quit IRC | 13:33 | |
| smarcet | ok | 13:33 |
| *** zz_dimtruck is now known as dimtruck | 13:33 | |
| fungi | zigo: jeblair is pretty close on a series of zuul patches to replace the limited bits of jenkins we still use | 13:33 |
| *** aysyd has joined #openstack-infra | 13:34 | |
| *** daemontool has quit IRC | 13:34 | |
| zigo | fungi: That's super nice. I just hope we'll be able to see build logs in real time again! :) | 13:34 |
| mordred | zigo: we do too | 13:35 |
| jhesketh | AJaeger: I'm not sure what you wanted to do with 308731... The only thing I can think of is group 3/4 being too greedy. Do you have an example of what you wanted to capture with it? Perhaps the easiest way would be to apply it again (perhaps to review-dev) and then tweak it | 13:35 |
| *** baumann has joined #openstack-infra | 13:38 | |
| *** jed56 has joined #openstack-infra | 13:38 | |
| *** chlong has joined #openstack-infra | 13:38 | |
| *** tonytan4ever has joined #openstack-infra | 13:38 | |
| rcarrillocruz | zigo: it seems we'll be able to see those via telnet (telnet is BACK) | 13:39 |
| fungi | well, plaintext tcp socket stream anyway | 13:39 |
| fungi | i doubt it will do telopt negotiation, 8-bit escapes, et cetera | 13:40 |
| *** zeih has joined #openstack-infra | 13:40 | |
| fungi | though if it needs them, i do have some experience writing a telnet option handler state engine in pure python ;) | 13:40 |
| Zara | rcarrillocruz: do you know anything about the current status of this? just going through tidying up old stories: https://storyboard.openstack.org/#!/story/153 | 13:41 |
| * rcarrillocruz suggests telnet towel.blinkenlights.nl as April's Fool day for zuul job logs stream | 13:41 | |
| rcarrillocruz | Zara: whoa, i totally forgot about that! | 13:42 |
| rcarrillocruz | so | 13:42 |
| pabelanger | rcarrillocruz: YES | 13:42 |
| rcarrillocruz | i started working on a prototype that used websockets with tornado to do that | 13:42 |
| rcarrillocruz | i'm not even sure if still have that around | 13:42 |
| *** thorst has joined #openstack-infra | 13:42 | |
| rcarrillocruz | Zara: is that subscription thing still a thing? | 13:42 |
| pabelanger | I had some good successes with zuulv2.5 last week. I need to loop back to it this afternoon to finish up the new zuul-slave DIB elements | 13:42 |
| rcarrillocruz | i honestly abandoned it, cos my employer told me to focus on other things, but given that i have more freedom now i wouldn't mind hacking on it | 13:43 |
| rcarrillocruz | pabelanger: :P | 13:43 |
| *** jpr has joined #openstack-infra | 13:43 | |
| pabelanger | now that telnet is back, we need to stand up bbs.openstack.org | 13:43 |
| pabelanger | LORD ftw | 13:43 |
| SotK | rcarrillocruz: it would be a cool and useful thing to have, but I don't think any work has been done on it since you last looked | 13:44 |
| Zara | rcarrillocruz: we have subscriptions in the api but I have no idea if it's the same thing, so probably not. :D | 13:44 |
| rcarrillocruz | so the idea would be to subscribe to storyboard over a websocket | 13:44 |
| rcarrillocruz | an events stream | 13:44 |
| *** thorst_ has quit IRC | 13:44 | |
| rcarrillocruz | so clients/users could do things on triggered events | 13:44 |
| *** ddieterly is now known as ddieterly[away] | 13:45 | |
| *** eharney has joined #openstack-infra | 13:45 | |
| rcarrillocruz | you should probably check with krotscheck for that | 13:45 |
| * SotK goes to find the spec he's definitely seen for this | 13:45 | |
| rcarrillocruz | if there's interest on that, i'm cool looking at it, just let me know | 13:45 |
| *** tosky has joined #openstack-infra | 13:46 | |
| rcarrillocruz | but i guess SB needs to stabilize and adress more common use cases | 13:46 |
| rcarrillocruz | that was more of a nice to have | 13:46 |
| *** zeih has quit IRC | 13:46 | |
| SotK | aha, I think it was this: https://review.openstack.org/#/c/105252/6/specs/storyboard_automated_pub_sub.rst | 13:46 |
| rcarrillocruz | ++, that was it | 13:46 |
| *** markvoelker has joined #openstack-infra | 13:46 | |
| Zara | I'll link it from the story so it doesn't get lost again | 13:47 |
| openstackgerrit | Merged openstack/requirements: Bump paramiko to version 2.0 https://review.openstack.org/314595 | 13:48 |
| * SotK is very interested in storyboard growing such functionality | 13:49 | |
| *** hdaniel_ has quit IRC | 13:50 | |
| *** fawadkhaliq has quit IRC | 13:50 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/shade: Don't fail getting flavors if extra_specs is off https://review.openstack.org/314640 | 13:50 |
| *** fawadkhaliq has joined #openstack-infra | 13:50 | |
| phschwartz | mordred: Shrews: ^^ | 13:51 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Add whitelist file to ostestr docs https://review.openstack.org/314775 | 13:52 |
| *** hongbin has joined #openstack-infra | 13:53 | |
| mordred | phschwartz: thanks! | 13:53 |
| *** ddieterly[away] is now known as ddieterly | 13:55 | |
| Zara | rcarrillocruz: if you want to hack on it, I'd also be happy about that. :) it might take longer for me to get to reviewing things if I'm prioritizing more general stability, but I feel like if we wait for it we'll never work on any of the nice-to-haves... so mainly depends on whether SB's ready enough to work on for that, I guess. | 13:55 |
| *** markvoelker has quit IRC | 13:55 | |
| *** dimtruck is now known as zz_dimtruck | 13:56 | |
| rcarrillocruz | yeah, i suggest we leave it unassigned and the SB team asses when it's sensible to get back to it? I think it may be the wiser thing, given the SB team is small enough | 13:57 |
| *** daemontool has joined #openstack-infra | 13:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix on OpenstackID job https://review.openstack.org/315031 | 13:58 |
| fungi | i can't find today's issue reported against seuptools, pip or bandersnatch... has anyone reported this somewhere upstream yet and if so, where? | 13:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Create a tripleo-ci project in JJB https://review.openstack.org/315062 | 13:59 |
| *** Na3iL has quit IRC | 13:59 | |
| *** xyang has quit IRC | 14:00 | |
| dstufft | fungi: https://github.com/pypa/pip/issues/3666 https://bitbucket.org/pypa/bandersnatch/pull-requests/20/fully-implement-pep-503-normalization/ | 14:00 |
| phschwartz | mordred: as to that change, I have tested it and it lets the os_server call work from ansible so it is a good fix till we decide on better handling of policy's blocking extra vars. | 14:01 |
| *** |-paul-| has quit IRC | 14:01 | |
| *** xyang has joined #openstack-infra | 14:01 | |
| *** ayoung has joined #openstack-infra | 14:01 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:01 | |
| *** links has joined #openstack-infra | 14:01 | |
| fungi | dstufft: oh, thanks! | 14:01 |
| fungi | i didn't think to look at open patches | 14:01 |
| fungi | though i finally found your discussion from earlier in scrollback where you said you submitted a patch | 14:02 |
| *** salv-orl_ has joined #openstack-infra | 14:04 | |
| *** kushal has quit IRC | 14:04 | |
| *** mhickey has joined #openstack-infra | 14:04 | |
| *** jed56 has quit IRC | 14:04 | |
| *** jed56 has joined #openstack-infra | 14:05 | |
| *** sree has joined #openstack-infra | 14:05 | |
| anteaya | mhickey: there you are | 14:06 |
| anteaya | hey folks I'd like you to meet martin hickey | 14:06 |
| mhickey | anteaya: Howdy! :-) | 14:06 |
| *** cdent has joined #openstack-infra | 14:06 | |
| anteaya | martin has been working on neutron things | 14:06 |
| anteaya | and now is told he is working on elk things | 14:06 |
| anteaya | clarkb: when you are online I'd like you to meet mhickey | 14:06 |
| *** salv-orlando has quit IRC | 14:06 | |
| *** mariojv has joined #openstack-infra | 14:06 | |
| anteaya | so it will take him a bit to get oriented to elk things | 14:07 |
| *** jed56 has quit IRC | 14:07 | |
| anteaya | but I told him he doesn't have to leave openstack to do his elk things | 14:07 |
| *** jed56 has joined #openstack-infra | 14:07 | |
| mariojv | hi infra - i'm here from #openstack-ironic and i'm curious about how the openstack proposal bot works. would it be possible to have it regenerate a sample configuration for the project in the same commit as a global requirements update? | 14:07 |
| anteaya | as we are doing elk things here and always are grateful for more opinions on the topic, as well as help | 14:07 |
| mariojv | the command is a simple tox -egenconfig | 14:07 |
| anteaya | mhickey: pabelanger and jesusaur also work on elk things as well | 14:08 |
| mhickey | anteaya: thanks. | 14:08 |
| anteaya | mhickey: thank you | 14:08 |
| dstufft | fungi: no problem :] | 14:08 |
| Zara | rcarrillocruz: sure, I can't promise it'll be a priority for me any time soon, but then priority is also affected by 'what are people able to work on (and interested in working on)?' ... since a bunch of people can say 'x is important' but that doesn't matter if nobody will resource it. | 14:08 |
| anteaya | mhickey: hope we can keep you working on openstack things while making management happy | 14:08 |
| *** sree__ has quit IRC | 14:08 | |
| *** jed56 has quit IRC | 14:08 | |
| mhickey | anteaya: sure! :D | 14:09 |
| *** jed56 has joined #openstack-infra | 14:09 | |
| anteaya | mhickey: wonderful | 14:09 |
| anteaya | mhickey: have you spent much time using our logstash.o.o implementation? | 14:09 |
| *** kushal has joined #openstack-infra | 14:10 | |
| mhickey | clarkb, pabelanger and jesusaurL look forward to chatting | 14:10 |
| *** fawadkhaliq has quit IRC | 14:10 | |
| mhickey | anteaya: nope; didn't know about it. | 14:10 |
| *** fawadkhaliq has joined #openstack-infra | 14:10 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Fix error in search due to multiple `title` columns in query https://review.openstack.org/315071 | 14:11 |
| fungi | mariojv: amrith started a thread on the -dev ml about publishing sample configs (note, embedding generated content into your source code is ugly and best avoided). dhellmann pointed out that there's now a mechanism to generate sample configuration and embed it in your sphinx docs when publishing them | 14:11 |
| *** xyang has quit IRC | 14:11 | |
| mariojv | thank you. I'll look at the mailing list thread. | 14:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Build and upload instack.raw image https://review.openstack.org/315072 | 14:12 |
| *** Na3iL has joined #openstack-infra | 14:13 | |
| *** daemontool has quit IRC | 14:13 | |
| mariojv | fungi: amrith: do either of you know the subject of the thread or have a link to it handy? if not i will search myself | 14:14 |
| *** Liuqing has quit IRC | 14:14 | |
| *** zz_dimtruck is now known as dimtruck | 14:14 | |
| fungi | mariojv: i don't recall it off hand, no. also dhellmann may just be able to point you to the config generation sphinx extension or whatever it is if he's around | 14:15 |
| *** psilvad has joined #openstack-infra | 14:15 | |
| *** jed56 has quit IRC | 14:15 | |
| *** xyang has joined #openstack-infra | 14:15 | |
| *** jed56 has joined #openstack-infra | 14:15 | |
| *** mixos has joined #openstack-infra | 14:15 | |
| *** fawadkhaliq has quit IRC | 14:15 | |
| dhellmann | fungi , mariojv : have a look at http://docs.openstack.org/developer/oslo.config/sphinxconfiggen.html and http://docs.openstack.org/developer/oslo.config/sphinxext.html and see how they are used in nova and glance respectively | 14:15 |
| *** jaosorior has quit IRC | 14:16 | |
| * amrith reads ... | 14:16 | |
| *** mixos_ has joined #openstack-infra | 14:16 | |
| amrith | mariojv, let me look for it | 14:17 |
| *** Swami has joined #openstack-infra | 14:17 | |
| mariojv | thank you amrith | 14:17 |
| *** Qiming_ has joined #openstack-infra | 14:17 | |
| *** Qiming has quit IRC | 14:18 | |
| *** jed56 has quit IRC | 14:18 | |
| *** zeih has joined #openstack-infra | 14:18 | |
| *** baumann1 has joined #openstack-infra | 14:18 | |
| *** jed56 has joined #openstack-infra | 14:18 | |
| krotscheck | EmilienM: It would. Once all the NPM things land I'll put your manifest on a VM and see what it looks like after running for a while. | 14:18 |
| *** rbrndt has joined #openstack-infra | 14:18 | |
| EmilienM | krotscheck: thanks, feel fee to take over my patch if I did something wrong. | 14:19 |
| EmilienM | free* | 14:19 |
| mariojv | amrith: i wouldn't worry about retrieving it now; we're not using oslo's config generator yet, so we would have to have that work finished before using this i believe | 14:19 |
| *** baumann has quit IRC | 14:19 | |
| krotscheck | You did nothing wrong. There just might be more to do :) | 14:19 |
| *** Liuqing has joined #openstack-infra | 14:19 | |
| *** baumann has joined #openstack-infra | 14:19 | |
| EmilienM | I'm willing to help, just let me know | 14:19 |
| krotscheck | Any infra-core around available to shell into mirror_update and give me a status on the output of the latest registry static update? | 14:20 |
| *** mixos has quit IRC | 14:20 | |
| krotscheck | It should be running in a screen. | 14:20 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Look for a passing periodic upgrades job before moving pin https://review.openstack.org/315075 | 14:20 |
| *** thorst has quit IRC | 14:21 | |
| *** dizquierdo has joined #openstack-infra | 14:21 | |
| amrith | mariojv, I'm not sure what fungi refers to ... | 14:21 |
| *** thorst has joined #openstack-infra | 14:21 | |
| amrith | I did post some comments on a review where someone wanted to use oslo config generator for Trove and it didn't work very nicely for reasons that are mostly Trove specific. | 14:22 |
| mariojv | ok - the documentation is sufficient to answer my questions | 14:22 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceph resize bug 1580625 https://review.openstack.org/315076 | 14:22 |
| openstack | bug 1580625 in OpenStack Compute (nova) "volume-backed resize intermittently fails with rbd imagebackend" [High,Confirmed] https://launchpad.net/bugs/1580625 | 14:22 |
| openstackgerrit | Dougal Matthews proposed openstack/requirements: Add tripleo-common to global requirements https://review.openstack.org/314078 | 14:22 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only https://review.openstack.org/315077 | 14:23 |
| *** baumann1 has quit IRC | 14:23 | |
| *** mariojv has left #openstack-infra | 14:23 | |
| *** rossella_s has quit IRC | 14:25 | |
| *** jamesmca_ has joined #openstack-infra | 14:25 | |
| *** thorst has quit IRC | 14:26 | |
| openstackgerrit | Graham Hayes proposed openstack/requirements: Block dnspython 1.13.0 https://review.openstack.org/315003 | 14:27 |
| *** sdake_ has quit IRC | 14:28 | |
| *** mixos_ has quit IRC | 14:28 | |
| *** sdake has joined #openstack-infra | 14:29 | |
| AJaeger | jhesketh: currently if a job fails the requirements check job, it does not show up in the list of job by Jenkins. ianw wanted to fix that with 308731. | 14:29 |
| *** anteaya has quit IRC | 14:30 | |
| *** amotoki has joined #openstack-infra | 14:30 | |
| *** jsavak has quit IRC | 14:30 | |
| fungi | amrith: oops, you're right, i was thinking of nikhil's thread on sample configs | 14:30 |
| fungi | mea culpa | 14:30 |
| *** jsavak has joined #openstack-infra | 14:30 | |
| amrith | fungi, no worries. | 14:31 |
| AJaeger | jhesketh: just do a change against nova that updates requirements file in such a way that the test will fail - and then check for that failure... Example: https://review.openstack.org/#/c/312722/ | 14:31 |
| *** daemontool has joined #openstack-infra | 14:31 | |
| nikhil | hemanthm: is r | 14:31 |
| *** jed56 has quit IRC | 14:31 | |
| AJaeger | jhesketh: you see the -1 by jenkins but only succeeding jobs and don't see the "Incompatible requirement found" | 14:31 |
| nikhil | Poc on config stuff | 14:31 |
| *** jed56 has joined #openstack-infra | 14:32 | |
| *** cdent has left #openstack-infra | 14:32 | |
| fungi | AJaeger: is there a retry for ianw's previous attempt to fix that? | 14:33 |
| AJaeger | ansiwen: for any change to requirements, some smoke testing is done - we cannot test all jobs on all repos ;( | 14:34 |
| AJaeger | fungi, I haven't seen one - just wanted to point out the problem to jhesketh since he was looking at the intel nvf problem which is in the same area... | 14:35 |
| *** xyang has quit IRC | 14:35 | |
| *** sdake has quit IRC | 14:35 | |
| *** pcrews has joined #openstack-infra | 14:36 | |
| *** xyang1 has joined #openstack-infra | 14:36 | |
| *** ajmiller has quit IRC | 14:36 | |
| *** Liuqing_ has joined #openstack-infra | 14:37 | |
| *** yamahata has joined #openstack-infra | 14:38 | |
| *** xyang has joined #openstack-infra | 14:38 | |
| *** asselin has joined #openstack-infra | 14:38 | |
| *** Liuqing has quit IRC | 14:38 | |
| pabelanger | AJaeger: I replied to your ML thread about bindep last night, let me know what you think when you have a moment | 14:39 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.log to 3.7.0 https://review.openstack.org/315086 | 14:39 |
| *** nadya has quit IRC | 14:39 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.messaging to 5.1.0 https://review.openstack.org/315087 | 14:40 |
| *** Liuqing_ has quit IRC | 14:40 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.middleware to 3.10.0 https://review.openstack.org/315088 | 14:40 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.privsep to 1.7.0 https://review.openstack.org/315089 | 14:40 |
| *** Liuqing has joined #openstack-infra | 14:41 | |
| AJaeger | pabelanger: if we install the test profile, anything that is not marked, gets installed as well. | 14:41 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.serialization to 2.6.0 https://review.openstack.org/315090 | 14:41 |
| *** mixos has joined #openstack-infra | 14:41 | |
| AJaeger | pabelanger: so, right now if we switch to test profile, it will not make a difference for package install at all. | 14:41 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.utils to 3.10.0 https://review.openstack.org/315091 | 14:41 |
| pabelanger | AJaeger: Yup, and I tested that locally too. | 14:41 |
| AJaeger | pabelanger: only if you add entries "package [non-test]", those would be excluded | 14:41 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for taskflow to 1.32.0 https://review.openstack.org/315092 | 14:41 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for tooz to 1.36.0 https://review.openstack.org/315093 | 14:42 |
| AJaeger | pabelanger: in the end I don't care whether we use test or not - as long as we document what we have and have a clear recommendation... | 14:42 |
| pabelanger | AJaeger: Right, by only point, is we have no way to test that today in the gate. So, if people want to use [test] they are happy too. We simply cannot enforce it | 14:42 |
| *** ajmiller has joined #openstack-infra | 14:42 | |
| *** zeih has quit IRC | 14:42 | |
| AJaeger | pabelanger: documentation https://review.openstack.org/#/c/314574/ | 14:42 |
| openstackgerrit | Julien Danjou proposed openstack-infra/release-tools: announce: make metatag optional https://review.openstack.org/315101 | 14:43 |
| AJaeger | pabelanger: correct, we cannot enforce it at all, we can just document it. And if we use test like https://review.openstack.org/314571 does it will just work | 14:43 |
| *** jaosorior has joined #openstack-infra | 14:43 | |
| AJaeger | pabelanger: so, I'm fine with fungi reviewing and saying how he likes it's done... | 14:44 |
| pabelanger | AJaeger: Yup, so I am happy to do what you have documented. | 14:44 |
| AJaeger | fungi, what's your opinion on this? | 14:44 |
| *** daemontool has quit IRC | 14:45 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Fix error in search due to multiple `title` columns in query https://review.openstack.org/315071 | 14:45 |
| AJaeger | stevemar: with publishing of the generated config file at http://docs.openstack.org/developer/keystone/sample_config.html do you still need to have the proposal job for the in-tree file? Or can we remove job and file? | 14:45 |
| *** Liuqing has quit IRC | 14:45 | |
| mordred | pabelanger, fungi: I'd like to restart nodepool with the fip cleaning patch applied - and concerns with doing that? | 14:46 |
| *** thorst has joined #openstack-infra | 14:46 | |
| *** jed56 has quit IRC | 14:47 | |
| pabelanger | mordred: nodepool-builder not affected? | 14:47 |
| *** jed56 has joined #openstack-infra | 14:47 | |
| mordred | pabelanger: no, it would be affected - we're in the middle of image build/upload aren't we? | 14:47 |
| *** zeih has joined #openstack-infra | 14:48 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack https://review.openstack.org/309383 | 14:48 |
| *** anteaya has joined #openstack-infra | 14:48 | |
| AJaeger | mordred: we should be finished, aren't we? We start earlier now... | 14:48 |
| pabelanger | mordred: Ya, won't be until later tonight until it finished. since we upload 56 images now | 14:48 |
| *** thorst has quit IRC | 14:48 | |
| AJaeger | ah... | 14:48 |
| mordred | nod | 14:48 |
| mordred | I'll wait then | 14:48 |
| *** thorst has joined #openstack-infra | 14:49 | |
| *** Liuqing has joined #openstack-infra | 14:49 | |
| pabelanger | mordred: if we land https://review.openstack.org/#/c/314331/ stack we can increase the number of upload workers to 12. Will need to land about 3 depends-on patches | 14:49 |
| sdague | could we get a promote of this - https://review.openstack.org/#/c/315106 ? | 14:49 |
| *** e0ne has quit IRC | 14:49 | |
| sdague | we had a new test that is cratering ceph | 14:49 |
| openstackgerrit | Beth Elwell proposed openstack-infra/project-config: Added openstack-server-publish-jobs to ironic-ui https://review.openstack.org/315107 | 14:49 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: update constraint for oslo.service to 1.10.0 https://review.openstack.org/315108 | 14:50 |
| zigo | pabelanger: mordred: https://review.openstack.org/#/c/311190/ <--- I've fixed deb-openstack-pkg-tools, so you can review this one again. | 14:50 |
| *** woodster_ has joined #openstack-infra | 14:50 | |
| mordred | pabelanger, clarkb: any specific reasons we haven't landed that series? | 14:50 |
| *** thorst has quit IRC | 14:50 | |
| *** jamesmca_ has quit IRC | 14:51 | |
| *** thorst has joined #openstack-infra | 14:51 | |
| pabelanger | mordred: clarkb: think we just need to schedule it at this point | 14:51 |
| mordred | gotcha | 14:51 |
| mordred | pabelanger: maybe let's do it when the current uploads finish | 14:51 |
| *** vincentll has joined #openstack-infra | 14:53 | |
| *** Qiming_ has quit IRC | 14:53 | |
| *** links has quit IRC | 14:54 | |
| *** thorst has quit IRC | 14:55 | |
| *** daemontool has joined #openstack-infra | 14:55 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/project-config: Added openstack-server-publish-jobs to ironic-ui https://review.openstack.org/315107 | 14:55 |
| *** jamesmca_ has joined #openstack-infra | 14:56 | |
| *** jamesmca_ has quit IRC | 14:56 | |
| *** jaosorior has quit IRC | 14:57 | |
| pabelanger | mordred: WFM | 14:57 |
| *** mtanino has joined #openstack-infra | 14:58 | |
| *** DevBox has joined #openstack-infra | 14:58 | |
| *** ddieterly has quit IRC | 14:58 | |
| *** claudiub has quit IRC | 14:59 | |
| *** ddieterly has joined #openstack-infra | 15:01 | |
| *** electrofelix has quit IRC | 15:01 | |
| *** baumann1 has joined #openstack-infra | 15:02 | |
| *** thorst_ has joined #openstack-infra | 15:03 | |
| *** sdake has joined #openstack-infra | 15:03 | |
| *** baumann has quit IRC | 15:04 | |
| *** electrofelix has joined #openstack-infra | 15:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Add api-ref jobs for Heat https://review.openstack.org/312726 | 15:06 |
| *** thorst_ has quit IRC | 15:07 | |
| dtantsur | hi folks! could someone approve a change to os-loganalyze: https://review.openstack.org/#/c/255231/ please? 5 months passed since its first and only +2 :) | 15:08 |
| *** thorst_ has joined #openstack-infra | 15:09 | |
| *** jamesmca_ has joined #openstack-infra | 15:09 | |
| *** matrohon has quit IRC | 15:10 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Test stack-update to new OPM https://review.openstack.org/315117 | 15:11 |
| *** sree has quit IRC | 15:11 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Magnum functional tests in Identity v3-only https://review.openstack.org/315118 | 15:12 |
| *** Liuqing has quit IRC | 15:12 | |
| *** jamesmca_ has quit IRC | 15:13 | |
| *** jistr has quit IRC | 15:13 | |
| *** thorst_ has quit IRC | 15:13 | |
| *** r-mibu has quit IRC | 15:14 | |
| *** r-mibu has joined #openstack-infra | 15:14 | |
| sdake | fungi ping re vmt | 15:14 |
| betherly | Hi there! have a project docs question. I have added openstack-server-publish-jobs to ironic-ui to create a link at http://docs.openstack.org/developer/ironic-ui (hopefully) https://review.openstack.org/#/c/315107/ | 15:14 |
| betherly | what happens next and how will i then directly edit the docs once they have been created? | 15:14 |
| sdake | betherly you create docs in your project's doc dir | 15:15 |
| sdake | the yget published by a tox publish job | 15:15 |
| sdake | its automagic once you have the infra in place in your project repo | 15:15 |
| sdake | like to see an example? | 15:15 |
| betherly | sdake: that would be great please! | 15:15 |
| betherly | so i have a docs directory with minimum stuff atm but can update. will that then publish with the patch i submitted on the next repo merge? | 15:16 |
| clarkb | fungi: you haven't retriggered that openstackid post job have you? | 15:16 |
| clarkb | fungi: smarcet I can do that shortly if not | 15:16 |
| sdake | https://github.com/openstack/kolla/blob/master/tox.ini#L34 | 15:16 |
| clarkb | sdake: the docs target is actually not used at all | 15:16 |
| sdake | betherly you need that - once that works locally your publish jobs should go to docs.oo/developer/ironic-ui | 15:17 |
| sdake | clarkb shows what I know ;) | 15:17 |
| AJaeger | betherly: we call 'tox -e venv python setup.py sphinx_build' in your repo and publish the content of doc/build/html to docs.openstack.org/developer/ironic-ui/ | 15:17 |
| *** esker has joined #openstack-infra | 15:17 | |
| *** yamamoto has quit IRC | 15:17 | |
| sdake | AJaeger thanks for the correction | 15:17 |
| clarkb | sdake: betherly https://governance.openstack.org/reference/cti/python_cti.html covers the testing interface for python projects which includes sphinx generation | 15:17 |
| *** salv-orl_ has quit IRC | 15:17 | |
| sdake | betherly you can test that locally by copyiing that tox.ini code | 15:17 |
| sdake | clarkb ack, just tyring to help ;) | 15:18 |
| *** yamamoto has joined #openstack-infra | 15:18 | |
| AJaeger | betherly: sory, typo, it's build_sphinx | 15:18 |
| AJaeger | betherly: so, if the that tox works, you should be set... | 15:18 |
| sdake | betherly also you need sphinx in requirements.txt or test-requirements, forget which | 15:19 |
| betherly | ok cool thanks everyone! | 15:19 |
| AJaeger | betherly: let me check for you and then review the change - if we're lucky, nothing else needs t obe done | 15:19 |
| *** zeih has quit IRC | 15:19 | |
| betherly | AJaeger: that would be amazing thanks :) | 15:20 |
| *** thorst_ has joined #openstack-infra | 15:20 | |
| *** salv-orlando has joined #openstack-infra | 15:21 | |
| *** winggundamth_ has joined #openstack-infra | 15:21 | |
| *** zeih has joined #openstack-infra | 15:21 | |
| fungi | AJaeger: pabelanger: seems fine to me. if projects want to annotate specific deps as test-only then this solution works fine, and if they don't want to go to the trouble of separating out test-only from other dependencies then they can just ignore it... however the tox env invocation probably needs to pass the test profile too for parity | 15:21 |
| fungi | mordred: no objection from me for a nodepool restart | 15:21 |
| *** fawadkhaliq has joined #openstack-infra | 15:22 | |
| fungi | other than image update interruptions, as noted | 15:22 |
| *** claudiub has joined #openstack-infra | 15:22 | |
| *** rcrit has quit IRC | 15:22 | |
| fungi | sdague: 315106 is still in check. did you want it enqueued and then promoted or were you wanting to wait for it to pass check first? | 15:22 |
| fungi | sdake: what did you want to discuss re: vmt things? | 15:23 |
| *** rcrit has joined #openstack-infra | 15:23 | |
| sdake | fungi we had discussed once a project achieves the vmt tagging, neew released (in this case neewton of kolla) could be covered by the ossa process | 15:23 |
| *** smarcet has quit IRC | 15:24 | |
| sdake | which totally works for me | 15:24 |
| *** jamesmca_ has joined #openstack-infra | 15:24 | |
| sdake | my quesiton is - we ar eadding a repo this cycle | 15:24 |
| fungi | clarkb: i have not retriggered post jobs for that openstackid commit since the latest job fix merged, no. feel free if you haven't already | 15:24 |
| pabelanger | fungi: agreed | 15:24 |
| sdake | fungi and it won't undergo threat analysis until the end of the cycle | 15:24 |
| AJaeger | betherly: everything looks fine. go ahead and add more documentation ;) | 15:24 |
| sdake | fungi it may not at all, as its an experimental repository | 15:24 |
| *** vhosakot has joined #openstack-infra | 15:24 | |
| AJaeger | fungi, yes, that's why I updated my swift change to use bindep test - and also updated the documentation | 15:25 |
| sdake | fungi just looking for some groudnrules around that point - if we don't release a final release because we don't think its ready, must we undergo TA for that repo as well? | 15:25 |
| betherly | AJaeger: thanks thats great! | 15:25 |
| *** thorst_ has quit IRC | 15:25 | |
| AJaeger | fungi, see https://review.openstack.org/314574 | 15:25 |
| sdague | fungi: yes | 15:25 |
| sdague | it's a skip for a test | 15:26 |
| *** kmartin has joined #openstack-infra | 15:26 | |
| sdague | so the safety level is high | 15:26 |
| *** esker has quit IRC | 15:26 | |
| sdague | and there is a project-config change coming to prevent that kind of issue happening in the future | 15:26 |
| openstackgerrit | Chris Denneen proposed openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options https://review.openstack.org/312121 | 15:26 |
| sdague | fungi: and, until it lands, a lot of stuff is going to be failing | 15:26 |
| *** esker has joined #openstack-infra | 15:27 | |
| sdague | fungi: like the nova change in gate slot #2 | 15:27 |
| *** jsavak has quit IRC | 15:27 | |
| fungi | sdague: i've started the enqueue, though it's currently behind zuul burning down the current events/results. as soon as the enqueue command returns i'll do a promote on it | 15:28 |
| *** tonytan_brb has joined #openstack-infra | 15:28 | |
| sdague | fungi: coolio, thanks | 15:28 |
| *** jsavak has joined #openstack-infra | 15:28 | |
| *** jamesmca_ has quit IRC | 15:28 | |
| fungi | sdake: the vulnerability:managed tag is per-deliverable, so the outcome depends somewhat on whether the new repo is part of the same covered deliverable or is a new, separate deliverable in the governance sense | 15:29 |
| *** jsavak has quit IRC | 15:30 | |
| clarkb | fungi: smarcet I have run the trigger job command, waiting now for job to run | 15:30 |
| sdake | fungi got it, so we could have kolla be a deliverable with vmt, an dkolla-kubernetes not be | 15:30 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Gate tempest on - gate-tempest-dsvm-full-devstack-plugin-ceph https://review.openstack.org/315124 | 15:30 |
| *** jsavak has joined #openstack-infra | 15:30 | |
| AJaeger | betherly: since you already gate on the doc job and can preview the contents (ever clicked on the gate-ironic-ui link?), everything is fine ;) | 15:30 |
| fungi | sdake: correct. also one could be covered starting with newton while another could be covered starting with ocata, for example | 15:31 |
| sdake | fantastic nes | 15:31 |
| sdake | news | 15:31 |
| sdake | thanks! | 15:31 |
| fungi | you're welcome! | 15:31 |
| *** amrith is now known as _amrith_ | 15:31 | |
| sdake | the docs indicate everything must be covered at oncei n the governance repo | 15:31 |
| *** tonytan4ever has quit IRC | 15:31 | |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config: Migrating liberasurecode and pyeclib projects https://review.openstack.org/314724 | 15:31 |
| *** jistr has joined #openstack-infra | 15:31 | |
| sdake | but maybe i misread them | 15:31 |
| sdake | its been a couple weeks since i looked at them (prior to summit) | 15:31 |
| AJaeger | pabelanger, fungi: Since you approved the bindep test change for project-config, could you review the docs change as well, please? https://review.openstack.org/314574 | 15:31 |
| *** armax has joined #openstack-infra | 15:32 | |
| betherly | AJaeger: oooooo! well thats exciting! | 15:32 |
| betherly | :D | 15:32 |
| fungi | sdake: "if some repos in a deliverable are in good enough shape to qualify, their vulnerability management could be held back by other repos in the same deliverable" http://governance.openstack.org/reference/tags/vulnerability_managed.html | 15:32 |
| sdake | got it - same deliverable | 15:32 |
| sdake | ok thanks :) | 15:32 |
| clarkb | mhickey: the current state of things is we are running logstash 2.0, elasticsearch 1.7.5, and kibana 3.something | 15:32 |
| *** madhu_ak has joined #openstack-infra | 15:33 | |
| fungi | sdake: the documentation tries to speak in terms of deliverable instead of project, because it's a deliverable tag not a project tag | 15:33 |
| *** _amrith_ is now known as amrith | 15:33 | |
| sdake | after 4 years of openstack, still learning my way around the system :) | 15:33 |
| fungi | we all are... we all are | 15:33 |
| *** jistr has quit IRC | 15:33 | |
| mhickey | clarkb: Hey. Ok. | 15:33 |
| *** jistr has joined #openstack-infra | 15:33 | |
| sdake | the sad part about that is im a "expert" :) | 15:33 |
| clarkb | mhickey: jesusaur has picked up the work to make our openstack logstash rules generically reconsumable in https://review.openstack.org/#/c/310046/ | 15:33 |
| clarkb | well not just that one change, but that repo | 15:33 |
| *** zeih has quit IRC | 15:34 | |
| sdake | fungi now that everyone has recovered from summit or mostly so, we will finish the job on ta | 15:34 |
| clarkb | which reminds me I need to finish reviewing those related changes and switch our logstash daemons over to those rulesets | 15:34 |
| sdake | our 4 hours of ta at summit went extremely well | 15:34 |
| clarkb | jesusaur: ^ any chance you will be around tomorrow? I think I will have time to devote to that tomorrow | 15:35 |
| fungi | sdake: absolutely. also tristanC can similarly answer questions (as can gmurphy if you find a channel he's in) | 15:35 |
| jesusaur | clarkb: yeah, i can make time tomorrow | 15:35 |
| *** ifarkas has quit IRC | 15:35 | |
| mhickey | clarkb: ok. so how long is infra using elk? | 15:35 |
| sdake | fungi cool thanks | 15:35 |
| clarkb | mhickey: years now. at laest 2 I think | 15:35 |
| clarkb | I would have to check git logs to give you an exact time period | 15:36 |
| *** xyang has quit IRC | 15:36 | |
| mhickey | clarkb: thats ok, a while is enough | 15:36 |
| dirk | dims: ping? | 15:36 |
| dims | dirk : pong | 15:36 |
| sdague | fungi: looks like event queue has drained | 15:37 |
| fungi | ttx: when you have a moment, trying to get control of https://launchpad.net/~openstack-vuln-mgmt in the hands of the current vmt members. i don't mind leaving you as the team owner but one or more of us probably need to be set to administrator | 15:37 |
| *** xyang has joined #openstack-infra | 15:37 | |
| *** mikelk has quit IRC | 15:37 | |
| mhickey | clarkb: what are use cases for using the stack? | 15:37 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove keystone-propose-config-updates https://review.openstack.org/315130 | 15:37 |
| sdague | also infra-core: https://review.openstack.org/#/c/315015/ - Intel NFV CI is effectively hiding Jenkins votes in Hide CI rollup, which is making reviewing code challenging. | 15:38 |
| fungi | sdague: 315106,1 should be at the front of the "integrated" gate queue as soon as the latest set of events/results are processed | 15:38 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for ceph resize bug 1580625 https://review.openstack.org/315076 | 15:38 |
| openstack | bug 1580625 in OpenStack Compute (nova) "volume-backed resize intermittently fails with rbd imagebackend" [High,Confirmed] https://launchpad.net/bugs/1580625 | 15:39 |
| fungi | and there it goes | 15:39 |
| sdague | fungi: awesome, thank you | 15:39 |
| jesusaur | mhickey: we collect all the test output and index that, we then automate searches of known bugs to find frequencies of e.g. race conditions | 15:39 |
| *** bpokorny has joined #openstack-infra | 15:39 | |
| *** armax has quit IRC | 15:40 | |
| mhickey | jesusaur: Hey. Thanks; kewl! | 15:40 |
| *** vincentll has quit IRC | 15:40 | |
| *** rraja has quit IRC | 15:41 | |
| *** winggundamth_ has quit IRC | 15:41 | |
| *** jamesmca_ has joined #openstack-infra | 15:41 | |
| *** xyang has quit IRC | 15:41 | |
| clarkb | right the idea is to make the large volume of logs generated by the test jobs more accessible | 15:41 |
| fungi | infra-root: once 315015 propagates to review.o.o, the ci results box will disappear from the gerrit web-ui until someone does a `sudo touch ~gerrit2/review_site/etc/GerritSiteHeader.html` on review.openstack.org. i'll try to catch it but if you don't, that's how to generally fix it | 15:41 |
| fungi | er, if _i_ don't | 15:42 |
| *** bpokorny has quit IRC | 15:43 | |
| ttx | fungi: added you and tristanC as admins, and removed myself from team | 15:43 |
| *** bpokorny has joined #openstack-infra | 15:43 | |
| mhickey | clarkb: nice :-) | 15:43 |
| *** ramishra has quit IRC | 15:44 | |
| openstackgerrit | Merged openstack-infra/shade: Don't fail getting flavors if extra_specs is off https://review.openstack.org/314640 | 15:44 |
| openstackgerrit | Madhumitha Viswanathan proposed openstack-infra/jenkins-job-builder: Add partial support for maven-release-plugin https://review.openstack.org/313196 | 15:45 |
| tristanC | ttx: thanks! | 15:45 |
| *** xyang has joined #openstack-infra | 15:45 | |
| *** ramishra has joined #openstack-infra | 15:46 | |
| sdague | fungi: thanks for the review on the js fix | 15:46 |
| *** rcernin has quit IRC | 15:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Use "bindep test" https://review.openstack.org/314571 | 15:48 |
| pabelanger | fungi: ack | 15:48 |
| daemontool | anyone available for a +A here https://review.openstack.org/#/c/314285/ it is about a new repo creation, thanks | 15:48 |
| clarkb | mhickey: are you looking to run one for yourself or interested in how we use it upstream or? | 15:48 |
| *** sshnaidm has quit IRC | 15:49 | |
| *** e0ne has joined #openstack-infra | 15:49 | |
| *** sshnaidm has joined #openstack-infra | 15:49 | |
| *** andymaier has quit IRC | 15:50 | |
| wznoinsk | infra cores... is there an official requirement for the CIs that would like to have voting rights? I hear two locations for resiliency etc but is there a checklist? | 15:50 |
| openstackgerrit | Dirk Mueller proposed openstack/requirements: Add review guideline about projects.txt updates https://review.openstack.org/315139 | 15:51 |
| wznoinsk | sdague: aren't cores interested in jenkins +1 changes so the issue intel nfv ci masking jenkins results in table is not actually that problematic? :-) | 15:51 |
| clarkb | wznoinsk: are you talking third party CI? if so that is entirely up to the projects they are voting on | 15:52 |
| *** armax has joined #openstack-infra | 15:53 | |
| clarkb | wznoinsk: if talking upstream CI then yes we require cloud redundancy for voting jobs | 15:53 |
| wznoinsk | clarkb: yes, 3rdparty, then if has gating rights requires redundancy? | 15:53 |
| mhickey | clarkb: going to be working upstream on elk in the next while. anteaya mentioned it is being used in OpenStack and then put me onto you folks! :) | 15:54 |
| clarkb | wznoinsk: third party cannot gate | 15:54 |
| fungi | ttx: thanks!!! | 15:54 |
| sdague | wznoinsk: on a jenkins +1 there are still potentially non voting jobs that failed, so being able to see that fact is important | 15:54 |
| wznoinsk | clarkb: I know, two thoughts in one sentence didn't go well here.. | 15:54 |
| *** Sukhdev has joined #openstack-infra | 15:54 | |
| openstackgerrit | Merged openstack-infra/system-config: Allow 3rd party CI's to have same pipeline names https://review.openstack.org/315015 | 15:55 |
| *** lakshmiS has joined #openstack-infra | 15:55 | |
| fungi | wznoinsk: are you asking whether a third-party ci system needs redundancy before projects will let it vote +1/-1 on changes? | 15:55 |
| mhickey | clarkb: so when I get ramped up, maybe I can be of help? | 15:55 |
| *** xyang has quit IRC | 15:55 | |
| *** xyang has joined #openstack-infra | 15:56 | |
| *** dtantsur is now known as dtantsur|afk | 15:56 | |
| wznoinsk | fungi: I think I take it as 'no' as per clarkb, I was trying to make another (shortcut) statement that if it gates it needs redundancy (for the OS infra gate only) | 15:56 |
| fungi | wznoinsk: if so, i don't know of any ecosystem-wide requirement for that. however, individual projects do get to decide which third-party ci systems vote on their changes and may impose such requirements on them as they see fit | 15:56 |
| wznoinsk | fungi: ok | 15:56 |
| clarkb | wznoinsk: I think our use of "gate" may be clashing here. In our use of zuul and gerrit the upstream CI is the only thing that gates. We require redundant clouds for those jobs to be voting | 15:57 |
| *** armax has quit IRC | 15:57 | |
| *** fitoduarte has joined #openstack-infra | 15:57 | |
| clarkb | wznoinsk: third party CI does not have this requirement at an openstack level and they do not gate. Individual projects (like nova or neutron) can allow third party CI to vote on changes and that is entirely up to them | 15:58 |
| wznoinsk | clarkb: and I understand only your OS infra zuul can gate which is the same thing? | 15:58 |
| fungi | wznoinsk: keep in mind that there are plenty of third-party ci systems which are run by the primary contributors for a given project and vote only on that project's repos (because perhaps they have some fundamental dependencies which cannot be tested upstream for various reasons), so we try to provide individual projects with as much levity as we can in these decisions | 15:58 |
| wznoinsk | fungi: clarkb: understood | 15:58 |
| *** xyang1 has quit IRC | 15:59 | |
| *** xyang1 has joined #openstack-infra | 15:59 | |
| fungi | however, as clarkb mentions, third-party ci systems don't have any ability to gate (i.e., automatically prevent changes from merging). only the upstream zuul gets to control that aspect | 16:00 |
| pabelanger | rcarrillocruz: crinkle: yolanda: was curious if there was any update on the IP address of infracloud? Or an etherpad tracking the progress | 16:00 |
| *** sridhar_ram_ has joined #openstack-infra | 16:01 | |
| *** jlanoux_ has quit IRC | 16:01 | |
| *** jsavak has quit IRC | 16:01 | |
| *** jsavak has joined #openstack-infra | 16:02 | |
| *** fawadkhaliq has quit IRC | 16:02 | |
| fungi | note that i don't particularly care whether there's a list of ip addresses, as long as they find their way into dns | 16:03 |
| fungi | as _that_ is the proper place for a list of ip addresses | 16:03 |
| *** cbader has quit IRC | 16:03 | |
| *** roxanaghe has joined #openstack-infra | 16:03 | |
| fungi | so don't feel obligated to do a bunch of unnecessary double-entry bookkeeping to track them as long as they get into dns | 16:04 |
| *** ffrank has quit IRC | 16:05 | |
| *** ddieterly is now known as ddieterly[away] | 16:06 | |
| *** dmk0202 has quit IRC | 16:07 | |
| crinkle | pabelanger: as far as I know rcarrillocruz is still working on it | 16:07 |
| *** nmagnezi has quit IRC | 16:07 | |
| *** mjblack has quit IRC | 16:08 | |
| jeblair | Released volume mirror.ubuntu successfully | 16:08 |
| jeblair | real 128m36.625s | 16:08 |
| jeblair | Released volume mirror.npm successfully | 16:08 |
| jeblair | real 567m34.443s | 16:08 |
| *** esker has quit IRC | 16:08 | |
| pabelanger | crinkle: thanks for the info | 16:08 |
| clarkb | jeblair: those are both from scratch to new afs02.dfw? | 16:08 |
| jeblair | yep | 16:09 |
| pabelanger | that seems faster | 16:09 |
| jeblair | so we've gone from like a week to 10 hours for a full replication of npm | 16:09 |
| mordred | jeblair: I think that's a good from-scratch | 16:10 |
| mordred | jeblair: npm is non-small | 16:10 |
| jeblair | yep | 16:10 |
| *** MarkAtwood has joined #openstack-infra | 16:11 | |
| pabelanger | https://review.openstack.org/#/c/314778/ also re-enabled vos release for mirror.centos too, since our initial sync is finished | 16:11 |
| clarkb | pabelanger: are we ready to update the ready script in nodepool to enable the centos mirror then? | 16:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Enable vos release for mirror.centos https://review.openstack.org/314778 | 16:12 |
| jeblair | clarkb, pabelanger: uncommented a line pabelanger missed, and approved ^ | 16:13 |
| *** abregman|afk has quit IRC | 16:13 | |
| fungi | mordred: we're able to mirror lots more isinteger() this way | 16:13 |
| *** gyee has joined #openstack-infra | 16:13 | |
| mordred | fungi: woot! | 16:14 |
| jeblair | krotscheck: the clean finished; i did a catchup run, and i'm vos releasing again (i expect it to have a lot to do and to take a while) | 16:14 |
| krotscheck | jeblair: Awesome! I'll come back... tomorrow? | 16:15 |
| *** cbader has joined #openstack-infra | 16:15 | |
| krotscheck | jeblair: I don't think that's accurate. The second run will have skipped any tarballs that it's already downloaded. | 16:16 |
| krotscheck | (to the above topic) | 16:16 |
| jeblair | #status log all afs mirror volumes have been moved to afs01.dfw and afs02.dfw (so they are no longer in ord) to speed up vos release times. all are in regular service using read-only replicas except for npm. | 16:16 |
| openstackstatus | jeblair: finished logging | 16:16 |
| jeblair | krotscheck: sorry, i meant the vos release will take a while. the catchup already finished. took only a few mins. | 16:16 |
| clarkb | pabelanger: is there a change up to reduce rax-ord max-servers yet? | 16:16 |
| krotscheck | Gotcha | 16:16 |
| clarkb | also the osic timeout attaching floating IP seems to be persistent, we should probably send them mail if that hasnt happened yet | 16:17 |
| jeblair | krotscheck: (but the vos release has all the changes that we made over the past 10+ hours) | 16:17 |
| krotscheck | Right. All the json files | 16:17 |
| krotscheck | Plus whatever tarballs we're missing. | 16:17 |
| *** ddieterly[away] is now known as ddieterly | 16:17 | |
| krotscheck | Honestly, I don't think it's going to take that long? | 16:17 |
| * krotscheck could be wrong | 16:17 | |
| krotscheck | Let me amend that. | 16:18 |
| krotscheck | "All the tarballs we're missing, plus whatever json files had bad versions in them (small)" | 16:18 |
| jeblair | krotscheck: yeah, though if it ended up touching files that didn't actually change (like, writing out all the json files regardless) could still take a while | 16:18 |
| jeblair | should be an upper bound of 10 hours though :) | 16:19 |
| clarkb | cloudnull: are you around? Seeing floating IP attachment take a long time (looks like close to an hour according to my logs) before the whole thing times out and is treated as an error in OSIC | 16:19 |
| clarkb | cloudnull: I can get instance UUIDs though it doesn't look like shade is currently logging the floating ip (or if it is I am not finding it iwth my grep yet) | 16:19 |
| krotscheck | :) | 16:20 |
| krotscheck | Alright, I'll pester you at 7PM :) | 16:20 |
| clarkb | oh sweet found it with better grepping | 16:20 |
| clarkb | cloudnull: 2016-05-11 16:15:48,128 ERROR shade: Timeout waiting for floating IP to become active. Floating IP 172.99.106.127:8e405996-111a-49da-8105-954327c3ba36 was created for server 055f7ffb-9e39-405e-982a-710fa462aff5 but is being deleted due to activation failure. is one example | 16:20 |
| *** armax has joined #openstack-infra | 16:21 | |
| pabelanger | jeblair: thanks | 16:21 |
| clarkb | mordred: ^ logs actually show it hapepning in bluebox too... | 16:21 |
| pabelanger | clarkb: no, not yet. I didn't know we wanted to reduce them | 16:21 |
| clarkb | mordred: I wonder if this is a shade issue and not necessarily a cloud issue (but only basing that on two clouds having same issue) | 16:22 |
| openstackgerrit | Merged openstack-infra/storyboard: Fix error in search due to multiple `title` columns in query https://review.openstack.org/315071 | 16:22 |
| pabelanger | clarkb: I was going to test the configure_mirror.sh script today | 16:22 |
| clarkb | pabelanger: well I am not sure we do, but we continue to hit that overquota error | 16:22 |
| *** mixos has quit IRC | 16:22 | |
| pabelanger | clarkb: Ya, same. Not sure why we are hitting it | 16:22 |
| *** claudiub has quit IRC | 16:23 | |
| *** hashar has quit IRC | 16:24 | |
| fungi | infra-root: just to follow up, it looks like our puppeted automation to touch GerritSiteHeader.html on a delay after hideci.js updates worked this time. maybe we got lucky? or maybe this is more robust in 2.11 | 16:24 |
| clarkb | mordred: there are indeed recent changes to shade in this area... | 16:24 |
| clarkb | mordred: https://review.openstack.org/#/c/308024/ /me is rereviewing to see if maybe this is the problem | 16:25 |
| clarkb | OH! | 16:25 |
| clarkb | previously we would leak the fip | 16:25 |
| clarkb | now we are bit a more explicit about things | 16:25 |
| clarkb | ok, so back to maybe clouds are broken (where clouds may also == neutron since mutiple clouds) | 16:27 |
| *** thorst_ has joined #openstack-infra | 16:27 | |
| *** baumann1 has quit IRC | 16:28 | |
| krotscheck | jeblair: I just ran `npm install` against horizon root with the new mirrors and it workred. | 16:28 |
| krotscheck | Well, with http://mirror.dfw.rax.openstack.org/npm/ | 16:29 |
| pabelanger | clarkb: so, this looks to be our default CentOS-Base.repo: http://paste.openstack.org/show/496776/ | 16:30 |
| jeblair | krotscheck: yeah, the read-only ones are are at the state we were in right before running clean, so i would expect most things to work | 16:30 |
| *** apoorvad has joined #openstack-infra | 16:30 | |
| pabelanger | clarkb: we are currently missing extras in rsync | 16:31 |
| krotscheck | jeblair: But they didn't until the cleaned json files went in. | 16:31 |
| *** apoorvad has quit IRC | 16:31 | |
| *** dizquierdo is now known as dizquierdo_afk | 16:31 | |
| *** thorst_ has quit IRC | 16:31 | |
| jeblair | krotscheck: i thought you said most json files were correct? | 16:31 |
| krotscheck | jeblair: They were, but horizon had bad dependencies which prevented it from working,. | 16:32 |
| fungi | per the -dev ml thread on "HW upgrade for tripleo CI" it sounds like we may need to keep an eye on nodepool in case we end up having to disable them in configuration | 16:32 |
| fungi | derekh: i take it from your e-mails that you're still on target to have it online again tomorrow? | 16:33 |
| fungi | the check-tripleo pipeline is at a ~5 hour backup right now with only 37 changes, but i expect that will grow quite a bit over the next day | 16:34 |
| derekh | fungi: yes that the plan, we should be back up and running some time tonight, the dcops guys are currently adding HW, once their done we should have everything back about 2 hours after | 16:34 |
| fungi | derekh: awesome. let us know if you hit any major issues that could cause significant delays to coming back online | 16:34 |
| derekh | fungi: will do | 16:35 |
| *** mhickey has quit IRC | 16:35 | |
| *** tosky has quit IRC | 16:36 | |
| zaro | morning | 16:36 |
| *** apoorvad has joined #openstack-infra | 16:36 | |
| *** tosky has joined #openstack-infra | 16:36 | |
| rcarrillocruz | hey folks, any reason why https://review.openstack.org/#/c/313520/ hasn't gone thru the gate? it got workflow, but i don't see it on the zuul page | 16:37 |
| rcarrillocruz | and it has no dependencies | 16:37 |
| *** baumann has joined #openstack-infra | 16:38 | |
| *** tphummel has joined #openstack-infra | 16:39 | |
| *** abregman|afk has joined #openstack-infra | 16:39 | |
| *** esikachev has quit IRC | 16:40 | |
| *** tellesnobrega_af is now known as tellesnobrega | 16:41 | |
| clarkb | pabelanger: we can just update the rsync command and resync right? | 16:42 |
| pabelanger | clarkb: yup, just checking that now | 16:43 |
| *** Jeffrey4l has quit IRC | 16:43 | |
| *** hashar has joined #openstack-infra | 16:43 | |
| clarkb | rcarrillocruz: it needs +1 jenkins, try rechecking it | 16:43 |
| *** mjblack has joined #openstack-infra | 16:44 | |
| rcarrillocruz | oh...probably there was a glitch when i pushed it, that's from last week... | 16:44 |
| rcarrillocruz | thanks | 16:44 |
| clarkb | fungi: still waiting for the openstackid-release-branch job to run but it is triggered | 16:44 |
| fungi | gate-tempest-dsvm-neutron-identity-v3-only-full sure seems to take a while | 16:46 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/requirements: Bump pyghmi version to 1.0.3 https://review.openstack.org/315161 | 16:46 |
| clarkb | fungi: and of course as soon as I say that it is done | 16:47 |
| clarkb | smarcet is no longer here though | 16:47 |
| clarkb | fungi: does puppet automatically pick up that new tarball or do we need to approve a change to deploy it to openstackid-dev? | 16:47 |
| fungi | did we get an updated branch tarball this time? | 16:47 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/314878 | 16:48 |
| krotscheck | Is configure_mirrors run on the release slave? | 16:48 |
| fungi | yeah, puppet retrieves and autodeploys the latest tarball each time it runs | 16:48 |
| *** hashar has quit IRC | 16:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add CentOS support to configure_mirror.sh https://review.openstack.org/315162 | 16:48 |
| clarkb | fungi: yes there are two new tarballs listed | 16:48 |
| pabelanger | clarkb: that should work ^ | 16:48 |
| pabelanger | clarkb: want to test it on a node first | 16:48 |
| clarkb | fungi: see http://tarballs.openstack.org/openstackid/?C=M;O=D | 16:48 |
| fungi | krotscheck: the long-lived job workers aren't built by nodepool and so don't get nodepool ready scripts run on them. they need their configuration managed by puppet where required | 16:49 |
| pabelanger | clarkb: holding a node now | 16:49 |
| clarkb | pabelanger: we need extras first though? | 16:49 |
| pabelanger | clarkb: Ya, doing that now | 16:49 |
| pabelanger | clarkb: will also add a Depends-On | 16:49 |
| krotscheck | fungi: Oh good, that makes my life easier. | 16:50 |
| *** claudiub has joined #openstack-infra | 16:50 | |
| *** Swami has quit IRC | 16:50 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/requirements: Bump pyghmi version to 1.0.3 https://review.openstack.org/315161 | 16:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Also mirror extras for centos-mirror-update.sh https://review.openstack.org/315163 | 16:51 |
| pabelanger | clarkb: once we land that, I'll grab the lock and manually rsync / vos release | 16:51 |
| fungi | sdague: looks like that tempest skip just merged | 16:52 |
| clarkb | we also have the occasional no valid host found in ovh-gra1, which I think means the nova scheduler couldn't place that instance | 16:52 |
| *** abregman|afk has quit IRC | 16:52 | |
| clarkb | pabelanger: ok | 16:52 |
| *** amotoki has quit IRC | 16:52 | |
| *** fawadkhaliq has joined #openstack-infra | 16:53 | |
| openstackgerrit | Dan Smith proposed openstack/requirements: Update oslo.versonedobjects to 1.9.1 and block 1.9.0 https://review.openstack.org/315164 | 16:53 |
| *** thorst_ has joined #openstack-infra | 16:53 | |
| jeblair | pabelanger: how large is extras? | 16:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Configure npm mirror for nodepool slaves. https://review.openstack.org/315166 | 16:54 |
| pabelanger | jeblair: just checking that now | 16:54 |
| *** asettle has quit IRC | 16:55 | |
| *** derekh has quit IRC | 16:55 | |
| openstackgerrit | venkatamahesh proposed openstack/requirements: Remove kolla-mesos from projects.txt https://review.openstack.org/313262 | 16:55 |
| *** javeriak has joined #openstack-infra | 16:56 | |
| *** baumann has quit IRC | 16:56 | |
| mordred | clarkb: good! so we're actually doing the log and delete properly | 16:56 |
| clarkb | mordred: I think we are | 16:56 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Magnum functional tests in Identity v3-only https://review.openstack.org/315118 | 16:57 |
| *** claudiub has quit IRC | 16:57 | |
| *** mat128 is now known as mat128|lunch | 16:57 | |
| *** psachin has quit IRC | 16:58 | |
| clarkb | mordred: we are functionally broken in better ways now \o/ | 16:58 |
| *** thorst_ has quit IRC | 16:58 | |
| mordred | clarkb: yay! | 16:58 |
| mordred | clarkb: it's all we can ask for | 16:58 |
| *** andreas_s has quit IRC | 16:58 | |
| AJaeger | sdague: could you review https://review.openstack.org/#/c/314795 , please - the TEMPEST_PLUGIN is unusual.. | 16:58 |
| clarkb | mordred: would be good if we can get someone from bluebox (jlk, timrc?) and someone from osic (cloudnull?) to look at the service logs for these FIPs | 16:58 |
| mordred | clarkb: agree | 16:58 |
| mordred | clarkb: I'd love to know more about the situatoin | 16:58 |
| clarkb | `grep 'is being deleted due to activation failure' /var/log/nodepool/debug.log` is the magic incantatiion for all the datas | 16:59 |
| clarkb | 169.45.92.x is bluebox and the rest is osic | 16:59 |
| *** javeriak_ has joined #openstack-infra | 16:59 | |
| openstackgerrit | Merged openstack/requirements: Remove Ceilometer from projects https://review.openstack.org/311989 | 16:59 |
| *** thorst_ has joined #openstack-infra | 16:59 | |
| mordred | clarkb: OH!!!!! | 17:01 |
| mordred | clarkb: there is still a leak | 17:01 |
| mordred | clarkb: 2016-05-11 16:56:05,471 ERROR shade: Timeout waiting for floating IP to become active. Floating IP 169.45.71.81:f27f46b9-0dfd-4e66-8253-89e3d5d1db31 was created for server 0dbe1ec7-e914-4530-864c-480c56288fc7 but is being deleted due to activation failure. | 17:01 |
| *** javeriak has quit IRC | 17:01 | |
| *** fernnest has quit IRC | 17:01 | |
| *** abregman has joined #openstack-infra | 17:01 | |
| mordred | clarkb: I show f27f46b9-0dfd-4e66-8253-89e3d5d1db31 in the leak list for bb | 17:01 |
| sdague | fungi: yep, thanks | 17:01 |
| clarkb | mordred: interesting | 17:01 |
| mordred | clarkb: so it seems that perhaps the delete call is failing | 17:01 |
| clarkb | ya | 17:01 |
| mordred | clarkb: we should log that too | 17:01 |
| *** jistr has quit IRC | 17:02 | |
| *** keedya has joined #openstack-infra | 17:02 | |
| clarkb | its like that floating ip goes off into lalala land | 17:02 |
| clarkb | timeout attaching, can't delete, boom leaked | 17:02 |
| mordred | yup | 17:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove keystone-propose-config-updates https://review.openstack.org/315130 | 17:03 |
| *** thorst_ has quit IRC | 17:04 | |
| pabelanger | jeblair: looks like 1.8GB for extras | 17:05 |
| *** nadya has joined #openstack-infra | 17:06 | |
| jeblair | pabelanger: i don't think i'd bother with a manual relase, that'll take just a few minutes | 17:06 |
| pabelanger | jeblair: okay, I'll monitor | 17:06 |
| *** dizquierdo_afk is now known as dizquierdo | 17:08 | |
| *** xyang has quit IRC | 17:09 | |
| *** xyang has joined #openstack-infra | 17:09 | |
| *** roxanaghe has quit IRC | 17:10 | |
| AJaeger | fungi, https://review.openstack.org/#/c/314240/ is a new project by krotscheck for Infra, please review. LGTM | 17:10 |
| *** xyang has quit IRC | 17:11 | |
| krotscheck | jeblair: Where would be the right place to document common mirror troubleshooting steps? | 17:11 |
| krotscheck | jeblair: For example-> Testing against storyboard-webclient, it appears that there's a missing package on the mirror. | 17:11 |
| *** fawadkhaliq has quit IRC | 17:12 | |
| *** yaume has quit IRC | 17:12 | |
| AJaeger | sdague: and similar https://review.openstack.org/#/c/299888 - could you put that on your review queue as well, please? | 17:12 |
| *** coreyob has quit IRC | 17:12 | |
| jlk | clarkb: mordred: so I'd like to get y'all off of Juno before digging too deep into why floating IPs are leaking. | 17:14 |
| pabelanger | looks like jenkins07 is down | 17:14 |
| mordred | jlk: ++ | 17:14 |
| pabelanger | going to start the clean up process | 17:14 |
| *** jed56 has quit IRC | 17:14 | |
| *** roxanaghe has joined #openstack-infra | 17:15 | |
| *** xyang has joined #openstack-infra | 17:15 | |
| *** wajdi has joined #openstack-infra | 17:15 | |
| clarkb | pabelanger: ok, you can run the ansible playbook and limit it to that host | 17:16 |
| clarkb | pabelanger: it runs against localhost so even with firewall in place should be fine | 17:16 |
| pabelanger | clarkb: okay, deleting jenkins07 nodes in nodepool first | 17:16 |
| wajdi | Would this be the proper place to ask about issues trying to dial in to a meeting using the conferencing system? | 17:16 |
| *** johnny__ has joined #openstack-infra | 17:16 | |
| clarkb | wajdi: it would | 17:16 |
| fungi | wajdi: definitely | 17:17 |
| pabelanger | I wonder if chan_sip is deadlocked again | 17:17 |
| wajdi | thanks clarkb . Last week I used hangouts and I got in fine. Currently trying and receiving "this is an invalid conference number" | 17:17 |
| AJaeger | thanks, fungi for review. Could you put the governance change also on your review queue? https://review.openstack.org/#/c/314239/ | 17:17 |
| wajdi | the number works, but the access code is not working | 17:17 |
| * AJaeger approves now krotscheck's new repo... | 17:17 | |
| krotscheck | woot | 17:17 |
| clarkb | wajdi: can we start with what conference tool you are using? maybe a link to the details? | 17:17 |
| fungi | wajdi: the number at https://wiki.openstack.org/wiki/Infrastructure/Conferencing is working for me | 17:17 |
| *** tqtran has joined #openstack-infra | 17:18 | |
| fungi | wajdi: are you trying to connect with sip instead? | 17:18 |
| wajdi | clarkb: fungi Certainly. I'm using Hangouts, dialing in to +1 512-808-5750 | 17:18 |
| jeblair | krotscheck: i don't think there are any common troubleshooting tasks | 17:18 |
| fungi | wajdi: i'm not sure what hangouts is | 17:18 |
| *** arxcruz has quit IRC | 17:18 | |
| wajdi | Google Hangouts | 17:18 |
| krotscheck | jeblair: Meta | 17:18 |
| pabelanger | clarkb: the restart_jenkins_masters.yaml right? | 17:19 |
| pleia2 | wajdi: it works for me in google hangouts | 17:19 |
| rcarrillocruz | so | 17:19 |
| fungi | wajdi: i didn't know google hangouts allowed you to dial into arbitrary conference call systems | 17:19 |
| wajdi | the number works. But the Conference code, e.g. 6555 does not | 17:19 |
| rcarrillocruz | infra-cloud update: | 17:19 |
| fungi | interesting | 17:19 |
| rcarrillocruz | the servers were racked up | 17:20 |
| pleia2 | wajdi: ah, let me try that | 17:20 |
| rcarrillocruz | on | 17:20 |
| rcarrillocruz | but iLOs are pulling from wrong VLAN | 17:20 |
| wajdi | fungi: I didn't think so either last week. But it did! :) | 17:20 |
| jeblair | 'that is not a valid conference number, please try again' is what i get if i dial in and do not enter a conference number | 17:20 |
| rcarrillocruz | talking with network folks now | 17:20 |
| wajdi | was pleasantly surprised | 17:20 |
| pleia2 | wajdi: I was able to dial into that call, people are talking | 17:20 |
| rcarrillocruz | i'll update when i get updates :-) | 17:20 |
| nibalizer | rcarrillocruz: neat | 17:20 |
| pleia2 | wajdi: use 6555# | 17:20 |
| pleia2 | on google hangouts | 17:20 |
| wajdi | @pleia2 thanks for looking in to that. | 17:21 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Swift functional tests in Identity v3-only https://review.openstack.org/313659 | 17:21 |
| pleia2 | rcarrillocruz: good luck :) | 17:21 |
| rcarrillocruz | lol | 17:21 |
| rcarrillocruz | ++ | 17:21 |
| wajdi | pleia2: definitely something really messed up on my end. I'm going to try jitsi. That should work. | 17:22 |
| *** ddieterly is now known as ddieterly[away] | 17:22 | |
| *** xyang has quit IRC | 17:22 | |
| pleia2 | wajdi: hope so | 17:22 |
| wajdi | thanks anyway! :) | 17:22 |
| *** simondodsley has joined #openstack-infra | 17:23 | |
| rcarrillocruz | who do i need to bribe for https://review.openstack.org/#/q/topic:cloud_launcher reviews ? | 17:23 |
| pabelanger | rcarrillocruz: thanks for the info | 17:23 |
| * rcarrillocruz shakes the review tin | 17:23 | |
| rcarrillocruz | pabelanger: yeah, even though is not a great info | 17:23 |
| rcarrillocruz | :D | 17:23 |
| *** jsavak has quit IRC | 17:23 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only https://review.openstack.org/313572 | 17:23 |
| pabelanger | wajdi: are you using a SIP client at all? | 17:23 |
| rcarrillocruz | i'll paste the exact thing: | 17:23 |
| pabelanger | sounds like DTMF is not reaching the far end | 17:23 |
| rcarrillocruz | "we need to adjust a few vlans on a switch, the problem is that the documentation RU of that switch and the physical don't match up, so we were working on confirming it's the right switch last i checked" | 17:24 |
| rcarrillocruz | ^ from HPE net engineer | 17:24 |
| *** f1ller is now known as filler | 17:24 | |
| pabelanger | okay, jenkins07 back online | 17:25 |
| *** electrofelix has quit IRC | 17:25 | |
| fungi | maybe they just need to drain some more water out of the linecards | 17:25 |
| pabelanger | and ~65 nodes back in play in the gate | 17:25 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only https://review.openstack.org/313588 | 17:26 |
| *** bpokorny has quit IRC | 17:26 | |
| *** esikachev has joined #openstack-infra | 17:27 | |
| *** bpokorny has joined #openstack-infra | 17:27 | |
| *** lucasagomes is now known as lucas-dinner | 17:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Add js-generator-openstack to OpenStack-Infra https://review.openstack.org/314240 | 17:27 |
| *** coreyob has joined #openstack-infra | 17:27 | |
| wajdi | jitsi was the winner | 17:27 |
| *** ociuhandu has quit IRC | 17:27 | |
| *** mriedem1 has joined #openstack-infra | 17:27 | |
| *** dizquierdo is now known as dizquierdo_afk | 17:28 | |
| *** Swami has joined #openstack-infra | 17:28 | |
| pabelanger | wajdi: which client wasn't working? | 17:28 |
| *** mriedem has quit IRC | 17:28 | |
| pabelanger | wajdi: check the dtmf settings and make sure rfc2833 is being use (out of band DTMF) | 17:28 |
| wajdi | pabelanger: For some reason Google Hangouts was just not working for me when trying to use the conference number. THe phone number worked, but not the conference number, e.g. 6555 | 17:28 |
| wajdi | (fyi...it works for me last week) | 17:29 |
| pabelanger | Ah | 17:29 |
| pabelanger | I'd say DTMF is coming from google inband, and is likely distorted | 17:29 |
| pabelanger | and asterisk cannot detect the tones | 17:29 |
| *** rkukura has quit IRC | 17:30 | |
| wajdi | @pabelanger that does make sense. I wonder if my VPN could be affecting this in any way | 17:30 |
| *** oanson has quit IRC | 17:30 | |
| pabelanger | wajdi: usually any sort of jitter on the line will cause DTMF not to work properly | 17:30 |
| pabelanger | default fix is always move to rfc2833 | 17:31 |
| *** mriedem1 is now known as mriedem | 17:31 | |
| *** e0ne has quit IRC | 17:31 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only https://review.openstack.org/315077 | 17:33 |
| *** openstackgerrit has quit IRC | 17:33 | |
| *** mat128|lunch is now known as mat128 | 17:33 | |
| *** openstackgerrit has joined #openstack-infra | 17:33 | |
| *** claudiub has joined #openstack-infra | 17:34 | |
| krotscheck | jeblair: Could you run this and paste me the output? It should add a new package that appears to be missing -> registry-static -d localhost -o /afs/.openstack.org/mirror/npm --blobstore afs-blob-store --hooks openstack-registry-hooks --one decompress | 17:36 |
| *** Na3iL has quit IRC | 17:36 | |
| *** fitoduarte has quit IRC | 17:37 | |
| *** sileht has quit IRC | 17:37 | |
| jeblair | krotscheck: you checked it's missing from the rw volume? | 17:39 |
| krotscheck | jeblair: No, however the package history goes back far enough that I'm fairly certain it should have gotten picked up by now. | 17:39 |
| krotscheck | jeblair: /afs/openstack.org/mirror/npm/d/decompress | 17:40 |
| jeblair | right, that's read-only; read-write is /afs/.openstack.org/mirror/npm/d/decompress and it also does not exist | 17:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Group DIB uploads by image name https://review.openstack.org/314231 | 17:41 |
| *** esikachev has quit IRC | 17:41 | |
| jeblair | krotscheck: http://paste.openstack.org/show/496785/ | 17:41 |
| *** esikachev has joined #openstack-infra | 17:41 | |
| *** matt-borland has quit IRC | 17:42 | |
| krotscheck | jeblair: That _looks_ right... | 17:42 |
| jeblair | krotscheck: that's kind of :( that the mirror didn't actually mirror it... | 17:42 |
| *** abregman has quit IRC | 17:42 | |
| *** Na3iL has joined #openstack-infra | 17:43 | |
| openstackgerrit | Merged openstack-infra/system-config: Enable vos release for mirror.centos https://review.openstack.org/314778 | 17:43 |
| krotscheck | jeblair: Yeah, I know. I'm trying to figure out if it may have been a bad regex in the blobstore. | 17:43 |
| krotscheck | jeblair: There's a bunch of packages named decompress-** which... I dunno, maybe did a thing? | 17:43 |
| krotscheck | Shouldn't have though | 17:43 |
| krotscheck | jeblair: did /afs/openstack.org/mirror/npm/d/decompress get created? I don't think it's synced to me yet. | 17:44 |
| *** sileht has joined #openstack-infra | 17:44 | |
| jeblair | krotscheck: that's read-only -- that happens with vos release, which means that it won't show up there until after the *next* vos release finishes (ie, not the one that is in progress) | 17:45 |
| jeblair | krotscheck: /afs/.openstack.org/mirror/npm/d/decompress exists now though | 17:45 |
| krotscheck | oh good | 17:45 |
| krotscheck | Well, no. not good, because why did the mirror not mirror that? | 17:45 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Bumps os-win version to 0.4.2 https://review.openstack.org/308506 | 17:46 |
| *** e0ne has joined #openstack-infra | 17:46 | |
| *** tosky has quit IRC | 17:46 | |
| *** ccamacho is now known as ccamacho|away | 17:46 | |
| *** ociuhandu has joined #openstack-infra | 17:51 | |
| asselin_ | anyone konw if we should use bandersnatch 1.10 or 1.10.0.1? https://bitbucket.org/pypa/bandersnatch/commits/branch/default | 17:52 |
| *** thorst_ has joined #openstack-infra | 17:53 | |
| asselin_ | I found the answer: 1.10.0.1 is the one to use: https://bitbucket.org/pypa/bandersnatch/commits/cdbe9e624e7863bf8e69427b79616aa777bc93d6?at=default | 17:54 |
| sdague | osic mirror issues - http://logs.openstack.org/30/314230/6/gate/gate-grenade-dsvm/f3b10ea/logs/grenade.sh.txt.gz#_2016-05-11_17_09_31_486 ? | 17:55 |
| * AJaeger waves good bye for today... | 17:58 | |
| *** e0ne has quit IRC | 17:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Also mirror extras for centos-mirror-update.sh https://review.openstack.org/315163 | 17:59 |
| *** austin81 has joined #openstack-infra | 18:02 | |
| openstackgerrit | Brian Rosmaita proposed openstack-infra/project-config: Add glance api-ref jobs https://review.openstack.org/315191 | 18:04 |
| *** sridhar_ram has joined #openstack-infra | 18:05 | |
| *** kmartin has quit IRC | 18:06 | |
| *** signed8bit is now known as signed8bit_Zzz | 18:06 | |
| *** Na3iL has quit IRC | 18:07 | |
| *** jamesmca_ has quit IRC | 18:08 | |
| *** keedya has quit IRC | 18:09 | |
| *** signed8bit_Zzz is now known as signed8bit | 18:09 | |
| *** pcrews has quit IRC | 18:10 | |
| *** pcrews_ has joined #openstack-infra | 18:10 | |
| pabelanger | clarkb: down to 5 image uploads, mainly because we only build fedora-23 and debian-jessie today, other failed because of missing repo. | 18:12 |
| clarkb | missing ubuntu repos? | 18:13 |
| clarkb | oh git repo | 18:13 |
| pabelanger | ya | 18:13 |
| *** rbrndt has quit IRC | 18:13 | |
| pabelanger | race condition with openstack/higgins | 18:13 |
| pabelanger | but, we should be able to land our nodepool-builder workers patch | 18:14 |
| *** IlyaG has joined #openstack-infra | 18:15 | |
| *** IlyaG has quit IRC | 18:16 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs https://review.openstack.org/313800 | 18:17 |
| dstufft | fungi: asselin_ noticed it already, but yea, bandersantch is released now with the fixes needed for that (and for not breaking when Warehouse releases!) | 18:17 |
| fungi | dstufft: thanks! i was just about to check back on that pr | 18:17 |
| fungi | i'll make sure our mirror updater is upgraded | 18:18 |
| clarkb | sdague: that package downloads currently. whatever it was seems transient | 18:18 |
| jeblair | fungi: unfortunately, our afs patch is still being ignored | 18:18 |
| clarkb | I can check apache logs t o see if they shed more light | 18:18 |
| jeblair | fungi: so it looks like we may need to legitimately fork bandersnatch | 18:18 |
| *** IlyaG has joined #openstack-infra | 18:18 | |
| JayF | https://review.openstack.org/#/c/313799 is a pretty easy review for ironic project-config, and is blocking us changing a devstack default; if someone has a moment and can review I'd appreciate it. | 18:19 |
| dstufft | jeblair: fwiw I think Christian is kind of overwhelmed with other stuff right now and struggling to find time to bandersnatch much | 18:20 |
| fungi | jeblair: oh, good point. i forgot we were running a lightweight fork already | 18:20 |
| asselin_ | dstufft, will 1.10.0.1 be shown in pypi, or is not not uploaded yet? | 18:20 |
| fungi | i'll hold off updating | 18:21 |
| asselin_ | we're going to pull the tag from hg | 18:21 |
| *** mixos has joined #openstack-infra | 18:21 | |
| mtreinish | jeblair: have you ever hit this before: http://paste.openstack.org/show/496793/ | 18:22 |
| dstufft | asselin_: I think that 1.10 was actually an OK release and 1.10.1 was not actually needed | 18:22 |
| dstufft | but I'm not sure | 18:22 |
| *** jsavak has joined #openstack-infra | 18:22 | |
| pabelanger | jeblair: fungi: looks like there is a new comment 2hours old: https://bitbucket.org/pypa/bandersnatch/pull-requests/16/add-option-to-dir-hash-index-files/diff | 18:22 |
| *** ddieterly[away] has quit IRC | 18:22 | |
| asselin_ | dstufft, he mentioned upload file failed https://bitbucket.org/pypa/bandersnatch/commits/cdbe9e624e7863bf8e69427b79616aa777bc93d6?at=default ...?? | 18:22 |
| dstufft | asselin_: yea, but then https://twitter.com/theuni/status/730450993444433920 | 18:23 |
| jeblair | dstufft: i sympathize, though i don't know what else to do. the patch is about 1.5 years old, has passing tests (including some added in response to code review), and we've been using it in production for more than 6 months. it's bitrotted sitting there twice. :( | 18:23 |
| asselin_ | dstufft, ok good to know. thanks | 18:23 |
| jeblair | mtreinish: neat. is that repeatable? | 18:24 |
| dstufft | jeblair: yea I don't have a good answer for you, just saying that I'm pretty sure Christian isn't like, purposely ignoring y'all or anything (I end up having to email him to get him to poke at even things to fix broken mirrors) | 18:25 |
| mtreinish | jeblair: I haven't tried to repeat it yet, gertty just crashed when I went to open a change | 18:25 |
| mtreinish | I'll try opening the same change again and see what happens | 18:26 |
| fungi | mtreinish: jeblair: i see that periodically too and keep meaning to try to recreate with debug enabled | 18:26 |
| mtreinish | jeblair: seems repeatable, crashed again | 18:26 |
| mtreinish | I forgot to run it in --debug, I'll do it now | 18:27 |
| fungi | i think it's a result of git gc removing some patchsets. if i manually re-fetch those commits from the origin, gertty stops breaking | 18:27 |
| *** xyang has joined #openstack-infra | 18:27 | |
| fungi | so might be as simple as an extra check in gertty to see if a commit it expects is actually present in the local repo, and retry to retrieve it (or error appropriately when in offline mode)? | 18:27 |
| jeblair | there are several things wrong though -- 1, all of our shas should have refs so that git gc doesn't remove them. 2, that is in a method that does exactly what fungi just describes | 18:28 |
| *** chem has quit IRC | 18:28 | |
| fungi | hrm | 18:28 |
| jeblair | so i wonder if there's a gitpython change that alters the behavior in that case | 18:28 |
| *** nadya has quit IRC | 18:28 | |
| jeblair | maybe it raises a new kind of exception | 18:28 |
| mtreinish | so running with --debug didn't output anything extra and the log is empty | 18:28 |
| clarkb | sdague: no obvious errors and the only GET on that url at that time 200'd | 18:28 |
| clarkb | sdague: the byte count looks off though | 18:29 |
| jeblair | mtreinish: what gitpython do you have? | 18:29 |
| sdague | clarkb: ok, I'll just assume blip | 18:29 |
| *** Sukhdev has quit IRC | 18:29 | |
| fungi | jeblair: mtreinish: GitPython==2.0.2 for me | 18:29 |
| mtreinish | jeblair: GitPython==2.0.2 | 18:30 |
| *** krotscheck_ has joined #openstack-infra | 18:30 | |
| *** fernnest has joined #openstack-infra | 18:30 | |
| *** Sukhdev has joined #openstack-infra | 18:31 | |
| *** krotscheck has quit IRC | 18:31 | |
| *** ddieterly has joined #openstack-infra | 18:32 | |
| *** jsavak has quit IRC | 18:32 | |
| *** krotscheck_ is now known as krotscheck | 18:33 | |
| *** xyang has quit IRC | 18:33 | |
| mtreinish | jeblair: is there anything else I should check before I get distracted by something else? | 18:33 |
| jeblair | mtreinish: i'm about to push up a patch | 18:33 |
| mtreinish | oh, ok cool | 18:33 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Handle ValueError on missing git commit https://review.openstack.org/315207 | 18:33 |
| *** jsavak has joined #openstack-infra | 18:34 | |
| *** sridhar_ram1 has joined #openstack-infra | 18:34 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only https://review.openstack.org/313588 | 18:34 |
| jeblair | mtreinish: ^ | 18:34 |
| *** sridhar_ram has quit IRC | 18:35 | |
| openstackgerrit | James E. Blair proposed openstack/gertty: Handle SNIMissingWarning requests exception https://review.openstack.org/315208 | 18:35 |
| bswartz | I'm having problems installing oslo packages -- I run my own devpi-server cache -- I saw the ML thread but I don't understand what I need to do to fix my own cache | 18:35 |
| *** tonytan_brb has quit IRC | 18:36 | |
| mtreinish | jeblair: seems to fix it, +1 (although it hangs while it's refetching the commits which was quite slow on the coffee shop wifi) | 18:36 |
| *** esikachev has quit IRC | 18:37 | |
| jeblair | mtreinish: yeah; if that's on a subscribed project, it likely means something went wrong already, so it's sort of a recovery operation at that point. if it happens a lot, maybe there's something else going on.... if it's a new change (like you clicked on a link or searched for it) then it's a defacto synchronous task anyway. | 18:38 |
| jeblair | mtreinish: note, that if you, say, removed a git repo from under gertty, it will do that for all the changes in that repo; instead, you can run once with "--fetch-missing-refs", and it will try to pull in anything missing at startup. | 18:38 |
| *** fitoduarte has joined #openstack-infra | 18:39 | |
| jeblair | (that is a slow operation, which is why it's not the default) | 18:39 |
| *** ddieterly is now known as ddieterly[away] | 18:40 | |
| fungi | bswartz: you need a new version of devpi i believe. there's a patch for it, though i don't know if there's been a release with that yet | 18:41 |
| *** rvasilets__ has joined #openstack-infra | 18:41 | |
| dstufft | devpi has been released | 18:41 |
| dstufft | afaik you just need a new devpi-common | 18:41 |
| *** weshay has quit IRC | 18:41 | |
| jeblair | Released volume mirror.npm successfully | 18:41 |
| jeblair | real 71m19.766s | 18:41 |
| jeblair | krotscheck: ^ | 18:41 |
| mtreinish | jeblair: I think it was probably related to my recent laptop migration. I copied the gerrty db (and yaml) but just recloned all the working git dirs (which was part of my normal install scripts) | 18:42 |
| bswartz | dstufft: ty I'll try to update | 18:42 |
| mtreinish | so it probably lost things it had locally checked out before and was surprised when they disappeared | 18:42 |
| jeblair | mtreinish: ah, then you will probably want to run --fetch-missing-refs when you are on a good network connection. until then, you'll have the slight delay when you open any old change | 18:42 |
| jeblair | mtreinish: but thanks, this case should have been handled, so i'm glad we found about the new behavior :) | 18:43 |
| jeblair | Released volume mirror.npm successfully | 18:43 |
| jeblair | real 1m12.752s | 18:43 |
| jeblair | krotscheck: ^ we're all caught up | 18:43 |
| *** e0ne has joined #openstack-infra | 18:44 | |
| *** ianychoi has quit IRC | 18:45 | |
| *** e0ne has quit IRC | 18:45 | |
| jeblair | krotscheck: i have released the mirror lock; so it should start running normally now | 18:46 |
| *** ianychoi has joined #openstack-infra | 18:47 | |
| *** pcrews_ has quit IRC | 18:47 | |
| *** ddieterly[away] is now known as ddieterly | 18:47 | |
| *** pcrews has joined #openstack-infra | 18:47 | |
| jeblair | #status log npm mirror sync finished; lock is released | 18:47 |
| openstackstatus | jeblair: finished logging | 18:47 |
| *** xyang has joined #openstack-infra | 18:47 | |
| pabelanger | jeblair: krotscheck: congrats! | 18:47 |
| *** sdake has quit IRC | 18:48 | |
| *** xyang has quit IRC | 18:49 | |
| *** tesseract has quit IRC | 18:51 | |
| *** pt_15 has joined #openstack-infra | 18:51 | |
| *** rbrndt has joined #openstack-infra | 18:52 | |
| *** xyang has joined #openstack-infra | 18:52 | |
| *** nadya has joined #openstack-infra | 18:55 | |
| *** xyang has quit IRC | 18:55 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Adding new repository trove-image-builder https://review.openstack.org/312805 | 18:56 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in heat https://review.openstack.org/315214 | 18:57 |
| *** vgridnev_ has joined #openstack-infra | 18:57 | |
| *** vgridnev_ has quit IRC | 18:57 | |
| *** ianychoi has quit IRC | 18:59 | |
| *** xyang has joined #openstack-infra | 18:59 | |
| *** tonytan4ever has joined #openstack-infra | 18:59 | |
| *** roxanaghe has quit IRC | 19:00 | |
| *** vgridnev_ has joined #openstack-infra | 19:00 | |
| pabelanger | okay, just confirmed: https://review.openstack.org/#/c/315162 centos mirror works correctly now that extras has been mirrored | 19:01 |
| openstackgerrit | Brian Rosmaita proposed openstack-infra/project-config: Add glance api-ref jobs https://review.openstack.org/315191 | 19:01 |
| clarkb | pabelanger: I can approve now | 19:02 |
| clarkb | pabelanger: is that ok? | 19:02 |
| pabelanger | clarkb: sure, I can monitor | 19:02 |
| krotscheck | jeblair: WOO! | 19:02 |
| krotscheck | OMG | 19:02 |
| * krotscheck does the ahppy dance | 19:02 | |
| krotscheck | Thanks so much for your hel on this, jeblair :) | 19:02 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add db api tests for subunit2sql (1/3) https://review.openstack.org/305030 | 19:03 |
| jeblair | krotscheck: the cron job just ran too | 19:03 |
| *** pcrews has quit IRC | 19:03 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add db api tests for subunit2sql (2/3) https://review.openstack.org/308147 | 19:03 |
| clarkb | pabelanger: btw I reall really really like that because we use rsync and centos doesn't aggressively delete things it means we can use the existing gpg trust chain | 19:03 |
| jeblair | krotscheck: so i think it's all set | 19:03 |
| krotscheck | niceeeeee | 19:03 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add db api tests for subunit2sql (3/3) https://review.openstack.org/309743 | 19:03 |
| *** baumann has joined #openstack-infra | 19:03 | |
| krotscheck | I'll send out the "Hey go check your projects" email, and then we can flip the switch early next week. | 19:03 |
| pabelanger | clarkb: Ya, it works pretty nice. Happy how it turned out. | 19:04 |
| *** keedya has joined #openstack-infra | 19:04 | |
| *** sambetts is now known as sambetts|afk | 19:04 | |
| pabelanger | clarkb: the last step is to write a simply python app, to validate we have all the files rsync'd (parse repodata info) before we do a vos release | 19:04 |
| *** pcrews has joined #openstack-infra | 19:05 | |
| *** esikachev has joined #openstack-infra | 19:06 | |
| clarkb | pabelanger: that is a confirmation that the index is complete? | 19:06 |
| clarkb | er s/complete/correct/ given the files on disk | 19:06 |
| pabelanger | clarkb: ya, a validation step. Make sure all files in repo index exist, we can even check md5 hash if needed | 19:07 |
| pabelanger | need to profile and see how long that takes | 19:07 |
| clarkb | tdurakov: any luck with libvirt cpu definitions on xenial? | 19:08 |
| *** jamesmca_ has joined #openstack-infra | 19:08 | |
| *** dtardivel has quit IRC | 19:09 | |
| *** weshay has joined #openstack-infra | 19:10 | |
| clarkb | fungi: any idea how to check what version of openstackid the dev server is running? | 19:11 |
| * clarkb checks puppetboard | 19:12 | |
| clarkb | http://puppetboard.openstack.org/report/openstackid-dev.openstack.org/636d64f984f8e96c3f31071a1a8e9e7921e85907 that must be it | 19:12 |
| *** jamesmca_ has quit IRC | 19:13 | |
| *** ddieterly is now known as ddieterly[away] | 19:13 | |
| *** ddieterly[away] is now known as ddieterly | 19:14 | |
| *** pvaneck has joined #openstack-infra | 19:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Add CentOS support to configure_mirror.sh https://review.openstack.org/315162 | 19:15 |
| pabelanger | clarkb: monitoring nodepool.log now | 19:15 |
| *** amrith is now known as _amrith_ | 19:16 | |
| *** ddieterly is now known as ddieterly[away] | 19:17 | |
| pabelanger | clarkb: keep forgetting, we need new images for nodepool scripts | 19:18 |
| clarkb | oh right | 19:18 |
| pabelanger | so, we're down to 2 DIB uploads on nodepool.o.o | 19:19 |
| *** vgridnev_ has quit IRC | 19:19 | |
| pabelanger | should be able to land the https://review.openstack.org/#/c/314332/ stack in the next 30mins | 19:19 |
| *** ianychoi has joined #openstack-infra | 19:19 | |
| fungi | clarkb: i think smarcet added some version query or maybe has it write out the version to disk or something? i'm not sure of the details | 19:20 |
| *** IlyaG has quit IRC | 19:21 | |
| pabelanger | clarkb: oh, and https://review.openstack.org/#/c/314812/ too please | 19:21 |
| *** davideagnello has quit IRC | 19:21 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in ceilometer https://review.openstack.org/315224 | 19:24 |
| clarkb | pabelanger: one thing on https://review.openstack.org/#/c/314329/2 | 19:24 |
| clarkb | otherwise stack lgtm and we can merge them all | 19:25 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Expose nodepool-builder worker settings https://review.openstack.org/314329 | 19:25 |
| pabelanger | clarkb: thanks! | 19:26 |
| clarkb | pabelanger: I approved ^, you will need to approve the child changes as their parents merge due to the depends on and no shared queue | 19:28 |
| pabelanger | clarkb: okay | 19:28 |
| pabelanger | still uploading to rax-iad | 19:29 |
| *** jsavak has quit IRC | 19:33 | |
| mtreinish | krotscheck: using the mirror to install openstack-health works for me locally. (although it was much slower) Just in case you were curious | 19:33 |
| openstackgerrit | Merged openstack-infra/system-config: Remove bare-precise from nodepool-builder logging https://review.openstack.org/314812 | 19:33 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Capitalize first characters in param docs https://review.openstack.org/306908 | 19:33 |
| *** jsavak has joined #openstack-infra | 19:33 | |
| *** rkukura has joined #openstack-infra | 19:34 | |
| *** jaosorior has joined #openstack-infra | 19:34 | |
| *** shardy has quit IRC | 19:36 | |
| fungi | mtreinish: slower? interesting. i wonder if you're hitting a closer cdn endpoint for lots of the deps when you don't pin to the mirror | 19:36 |
| *** rkukura has quit IRC | 19:36 | |
| *** bnemec has quit IRC | 19:36 | |
| lifeless | fungi: yes | 19:37 |
| lifeless | fungi: pypi is fast.ly mirrored | 19:37 |
| *** BobH_ has joined #openstack-infra | 19:37 | |
| *** DevBox has quit IRC | 19:37 | |
| *** lakshmiS_ has joined #openstack-infra | 19:38 | |
| fungi | lifeless: i think this was about the npm mirror, not the pypi mirror | 19:38 |
| fungi | so presumably github cdn endpoints or somethintg | 19:38 |
| mtreinish | fungi: that I can't say, it was just an anecdotal observation. Normally when I do a fresh npm install on it it's very fast that it doesn't make sense to go do something else | 19:38 |
| *** ddieterly[away] is now known as ddieterly | 19:38 | |
| *** dmk0202 has joined #openstack-infra | 19:38 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in magnum https://review.openstack.org/315229 | 19:39 |
| *** jamesmca_ has joined #openstack-infra | 19:39 | |
| fungi | mtreinish: local caching which was invalidated the first time you installed from the new location maybe? do subsequent reinstalls with the mirror go faster? | 19:39 |
| mtreinish | fungi: but in this case it took several minutes | 19:39 |
| *** matrohon has joined #openstack-infra | 19:39 | |
| fungi | i guess we'll have some more objective data once we have horizon functional jobs running against the mirror | 19:39 |
| *** e0ne has joined #openstack-infra | 19:39 | |
| *** lakshmiS has quit IRC | 19:39 | |
| mtreinish | fungi: maybe, but I was thinking more about when I was setting up my new laptop last week | 19:40 |
| mtreinish | I'll test things more when I go home | 19:40 |
| fungi | mtreinish: another possibility is cold cache on the mirror | 19:40 |
| pabelanger | fungi: mtreinish: that's what I was thinking | 19:40 |
| fungi | since our "mirrors" are just caches | 19:40 |
| fungi | so cache misses take teh slow boat over afs/udp | 19:40 |
| *** e0ne has quit IRC | 19:41 | |
| jeblair | oh yeah, cold caches on our mirrors are quite slow. but that doesn't last long around these parts. | 19:41 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in barbican https://review.openstack.org/315231 | 19:41 |
| daemontool | fungi, jeblair if you have any spare minutes please, need +A, new repo creation, so tomorrow the Team can keep working on it https://review.openstack.org/#/c/314285/ | 19:42 |
| krotscheck | mtreinish: Thanks! That's a good data point. | 19:42 |
| *** bnemec has joined #openstack-infra | 19:44 | |
| smcginnis | Anyone else getting stacking failures finding oslo.context 2.3.0? | 19:45 |
| openstackgerrit | Itisha Dewan proposed openstack-infra/project-config: Adding library Delimiter to the CI system https://review.openstack.org/315232 | 19:45 |
| *** jsavak has quit IRC | 19:46 | |
| *** jsavak has joined #openstack-infra | 19:46 | |
| fungi | smcginnis: pip 8.1.2? using a pypi mirror (maybe via devpi)? | 19:47 |
| *** rkukura has joined #openstack-infra | 19:48 | |
| smcginnis | fungi: Looks like at least on one machine still using 8.0.2. Not local mirror, direct to pypi. | 19:49 |
| *** r-daneel has joined #openstack-infra | 19:49 | |
| smcginnis | INSTALLED: 2.2.0 | 19:49 |
| smcginnis | LATEST: 2.0.0 | 19:49 |
| smcginnis | fungi: On pypi I can go direct to the package, so it's there: https://pypi.python.org/pypi/oslo.context/2.3.0 | 19:49 |
| smcginnis | fungi: But a search does not show it: https://pypi.python.org/pypi?%3Aaction=search&term=oslo.context&submit=search | 19:50 |
| smcginnis | Max is 2.0.0. | 19:50 |
| fungi | smcginnis: doesn't sound familiar in that case. maybe the devstack devs in #opsnstack-qa have more of an idea as to what's going on with ut | 19:50 |
| smcginnis | I did get confirmation from at least one other person that they are getting the same thing. | 19:50 |
| fungi | smcginnis: don't trust the "search" feature on pypi.python.org, it's long broken | 19:50 |
| fungi | and pip doesn't depend on pypi search, it does its own index enumeration | 19:50 |
| smcginnis | fungi: Yeah, just interesting that the pip search for the package is reflecting the same thing. | 19:50 |
| *** jaosorior has quit IRC | 19:51 | |
| fungi | smcginnis: weird, http://codesearch.openstack.org/?q=oslo.context.*2.3.0 says it's in the constraints list | 19:52 |
| smcginnis | fungi: Yep. My CI is failing to stack because of this. | 19:52 |
| fungi | we also have tarballs/wheels for it archived at http://tarballs.openstack.org/oslo.context/ | 19:52 |
| fungi | oh, https://pypi.python.org/pypi/oslo.context also shows 2.3.0 as available | 19:53 |
| smcginnis | Just got a report of the same situation with oslo.config | 19:53 |
| *** tphummel has quit IRC | 19:53 | |
| smcginnis | fungi: Yeah, the package is out there. The search just reflects the pip results of a max of 2.0.0. | 19:53 |
| smcginnis | Pypi's index is messed up maybe? | 19:53 |
| *** rkukura_ has joined #openstack-infra | 19:54 | |
| smcginnis | Perhaps related: https://sourceforge.net/p/pypi/support-requests/626/ | 19:54 |
| clarkb | have a paste of the output of a failed pip install? | 19:54 |
| *** tphummel has joined #openstack-infra | 19:54 | |
| mriedem | AJaeger: should new jobs be using xenial nodes? https://review.openstack.org/#/c/313588/4/jenkins/jobs/projects.yaml | 19:54 |
| fungi | yeah, sometimes fastly negative-caches some urls for no good reason | 19:55 |
| bswartz | Updating my devpi-server cache to the latest version solved my oslo.config install issues | 19:55 |
| bswartz | I also did a rm -rf ~/.devpi | 19:55 |
| smcginnis | clarkb: http://paste.openstack.org/show/496809/ | 19:55 |
| fungi | smcginnis said he's not using devpi though, and is also on older pip (8.0.2) so presumably not related | 19:55 |
| smcginnis | fungi: Also tried updating pip to latest, same results. | 19:56 |
| clarkb | the from versions list is empty | 19:56 |
| clarkb | I think that means it doesnt see the package at all | 19:56 |
| *** HeOS has quit IRC | 19:57 | |
| smcginnis | clarkb: pip search output: http://paste.openstack.org/show/496810/ | 19:57 |
| *** rkukura has quit IRC | 19:57 | |
| *** rkukura_ is now known as rkukura | 19:57 | |
| clarkb | ya ignoring search because install is the best data for install | 19:57 |
| *** mixos has quit IRC | 19:57 | |
| *** ccamacho|away has quit IRC | 19:58 | |
| *** thorst_ has quit IRC | 19:59 | |
| *** ccamacho has joined #openstack-infra | 19:59 | |
| *** vilobhmm11 has joined #openstack-infra | 20:00 | |
| clarkb | I cant reproduce locallyfwiw | 20:00 |
| smcginnis | clarkb: DO you have a local cache? | 20:00 |
| *** itisha has joined #openstack-infra | 20:01 | |
| *** thorst_ has joined #openstack-infra | 20:01 | |
| clarkb | no | 20:01 |
| *** yamahata has quit IRC | 20:01 | |
| smcginnis | hmmm | 20:01 |
| clarkb | smcginnis: the warnings from pip searchimply you are not talki g to pypi | 20:01 |
| clarkb | you should not get the sni warning against pypi iirc | 20:02 |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/system-config: Add zuul_launcher to openstack_project and run on jenkinsXX hosts. https://review.openstack.org/311165 | 20:02 |
| vilobhmm11 | clarkb, fungii, anteaya, AJaegar : working on the effort to create a new lib for cross-project quota enforcement and here is the patch that itisha created to add "delimiter" to CI system https://review.openstack.org/#/c/315232/1 can someone please have a look. This will help us to move quickly with this cross project effort | 20:02 |
| vilobhmm11 | itisha, nikhil : ^^ | 20:03 |
| nikhil | vilobhmm11: we also have CP spec ppl can refer | 20:03 |
| *** jsavak has quit IRC | 20:03 | |
| *** salv-orl_ has joined #openstack-infra | 20:03 | |
| vilobhmm11 | sure CP spec for reference : https://review.openstack.org/#/c/284454/ | 20:04 |
| *** jsavak has joined #openstack-infra | 20:04 | |
| *** sdake has joined #openstack-infra | 20:05 | |
| *** thorst_ has quit IRC | 20:06 | |
| *** salv-orlando has quit IRC | 20:06 | |
| mtreinish | fungi, krotscheck: I ran things locally again (with time this time) http://paste.openstack.org/show/496812/ | 20:08 |
| mtreinish | it definitely wasn't as slow as the first time I ran it (also the it was slower than I remembered without the mirror) | 20:09 |
| *** hashar has joined #openstack-infra | 20:09 | |
| fungi | mtreinish: 7 seconds difference i would believe | 20:09 |
| krotscheck | mtreinish: I'm secretly having everyone run this so it warms up the AFS caches. | 20:09 |
| krotscheck | So a lot of that initial lag is probably the cache warming up on dfw | 20:09 |
| *** daemontool has quit IRC | 20:09 | |
| fungi | our mirror servers aren't necessarily well-connected or high-performance either compared to what github et al are likely fronted with | 20:10 |
| krotscheck | With that in mind, npmjs.org has its own CDN, so it's expected that using them will probably be faster. | 20:10 |
| fungi | ahh, so there is a specific cdn for most npm stuff? even less surprising result in that case | 20:10 |
| krotscheck | fungi: Yep | 20:11 |
| *** e0ne has joined #openstack-infra | 20:11 | |
| krotscheck | fungi: I expect that number to be on-par or better when we're running it on a node in the same infra cluster. | 20:11 |
| fungi | i think "takes 15% longer than npmjs.org" is a perfectly reasonable performance expectation | 20:11 |
| *** vgridnev has joined #openstack-infra | 20:12 | |
| mtreinish | yeah, that's probably fine | 20:12 |
| *** vgridnev has quit IRC | 20:12 | |
| fungi | and yes, most actual use of these mirrors will be over sub-milisecond latency links | 20:12 |
| clarkb | in region should be fast on warm cache | 20:12 |
| clarkb | ya that | 20:12 |
| mtreinish | krotscheck: well now I'm gonna run it against ord and other regions :) | 20:13 |
| openstackgerrit | Itisha Dewan proposed openstack-infra/project-config: Adding library Delimiter to the CI system https://review.openstack.org/315232 | 20:13 |
| krotscheck | mtreinish: Yay! | 20:13 |
| *** e0ne has quit IRC | 20:13 | |
| * fungi wonders if a cold cache is more like gazpacho or more like borscht | 20:13 | |
| krotscheck | mtreinish, fungi: I'm looking for packages that didnt' get synced. Apparenlty the mirroring script has problems with those :/ | 20:13 |
| krotscheck | fungi: Gazpacho, I would say. Because you know there's a bunch of vegetables there, but you can't identify them. | 20:14 |
| *** daemontool has joined #openstack-infra | 20:14 | |
| openstackgerrit | Merged openstack/requirements: Ignore more build temp dirs https://review.openstack.org/251604 | 20:14 |
| fungi | wfm | 20:14 |
| krotscheck | So, now that i have javascript things in infra again that I'm the main developer on, who wants to volunteer to help land patches? :) | 20:15 |
| pabelanger | clarkb: okay, image uploads done. Starting the +A on nodepool-builder patches | 20:15 |
| openstackgerrit | Merged openstack-infra/glean: Enable multiple VLANs per physical interface https://review.openstack.org/283726 | 20:16 |
| *** _amrith_ is now known as amrith | 20:19 | |
| *** xyang has quit IRC | 20:19 | |
| mtreinish | krotscheck: just ord, i'm too impatient to wait ~6min again: http://paste.openstack.org/show/496816/ | 20:19 |
| openstackgerrit | Merged openstack-infra/nodepool: Expose nodepool-builder workers settings https://review.openstack.org/294339 | 20:23 |
| hongbin | AJaeger: Hi, I wonder if you can add me to these two gerrit group: higgins-core, higgins-release | 20:26 |
| *** mjblack has quit IRC | 20:27 | |
| *** earlephilhower has quit IRC | 20:28 | |
| *** annegentle has joined #openstack-infra | 20:28 | |
| *** kgiusti has left #openstack-infra | 20:30 | |
| openstackgerrit | greghaynes proposed openstack-infra/glean: Enable support for configuring bonded interfaces https://review.openstack.org/293648 | 20:30 |
| *** earlephilhower has joined #openstack-infra | 20:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Group DIB uploads by image name https://review.openstack.org/314231 | 20:32 |
| pabelanger | clarkb: okay, that should happy in the gate now^ | 20:33 |
| *** kien-ha has joined #openstack-infra | 20:33 | |
| *** esikache1 has joined #openstack-infra | 20:33 | |
| *** ddieterly is now known as ddieterly[away] | 20:34 | |
| *** kien-ha has quit IRC | 20:34 | |
| *** jamesmca_ has quit IRC | 20:34 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Expose nodepool-builder worker settings https://review.openstack.org/314329 | 20:34 |
| cloudnull | clarkb: was just reading scroll back im missed the ping and have been locked up in meetings most of the day sorry, is the floating IP situation figued out? | 20:34 |
| jeblair | pabelanger: oh, did we decide not to do the other approach? | 20:34 |
| clarkb | cloudnull: it is not | 20:35 |
| cloudnull | RE: OSIC problems ... | 20:35 |
| clarkb | cloudnull: they seem to periodically timeout on attach then fail to delete | 20:35 |
| pabelanger | jeblair: no, I think upload right away after build is the way. I just don't know how to do that yet :) | 20:35 |
| clarkb | cloudnull: did you see the example I pasted? should have uuids for ip and server would be great to see why that happened | 20:35 |
| *** esikachev has quit IRC | 20:35 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Expose nodepool-builder workers settings https://review.openstack.org/314331 | 20:36 |
| *** vilobhmm11 has quit IRC | 20:36 | |
| *** xyang has joined #openstack-infra | 20:36 | |
| cloudnull | clarkb: i see the log output from shade . do you have the like for the paste? | 20:37 |
| * cloudnull catching up | 20:37 | |
| *** kien-ha has joined #openstack-infra | 20:37 | |
| *** baumann has quit IRC | 20:37 | |
| *** ddieterly[away] is now known as ddieterly | 20:37 | |
| *** mjblack has joined #openstack-infra | 20:37 | |
| clarkb | cloudnull: wasnt a paste was just here in irc | 20:38 |
| *** roxanaghe has joined #openstack-infra | 20:38 | |
| *** xyang has quit IRC | 20:38 | |
| cloudnull | ok, i see that log output, im logging into the env now and will start poking around | 20:38 |
| pabelanger | jeblair: If I understand, the issue is on 1635 of nodepool.py. I guess we need to refactor waitForBuiltImages() to waitForBuildImageThenUpload() | 20:39 |
| openstackgerrit | Merged openstack-infra/system-config: Switch to 12 upload-workers for nodepool https://review.openstack.org/314332 | 20:39 |
| cloudnull | and will post back regarding issues i find | 20:39 |
| *** baumann has joined #openstack-infra | 20:40 | |
| *** thorst_ has joined #openstack-infra | 20:41 | |
| clarkb | cloudnull: great thank you | 20:41 |
| jeblair | pabelanger: yes -- but you'll have a pool of things you're waiting for and won't know which will finish first (assuming there is more than one builder). so i would just put each image build+upload into its own thread. | 20:41 |
| *** ayoung has quit IRC | 20:41 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:42 | |
| *** dimtruck is now known as zz_dimtruck | 20:42 | |
| *** thorst__ has joined #openstack-infra | 20:42 | |
| *** pcaruana has quit IRC | 20:42 | |
| pabelanger | jeblair: okay, that makes sense. Going to take me a few tries to get it, threading not my strongest suit | 20:43 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/devstack-gate: Add Higgins to PROJECTS list https://review.openstack.org/315248 | 20:43 |
| jeblair | pabelanger: a few lines up, updateImage launches a thread for each snapshot updater | 20:43 |
| *** rcernin has joined #openstack-infra | 20:44 | |
| *** thorst___ has joined #openstack-infra | 20:44 | |
| *** icey has quit IRC | 20:44 | |
| pabelanger | jeblair: Ya, I see that now. Okay, I'll use SnapshotImageUpdater as a reference | 20:45 |
| *** thorst_ has quit IRC | 20:45 | |
| *** thorst_ has joined #openstack-infra | 20:45 | |
| *** icey has joined #openstack-infra | 20:46 | |
| *** _ari_ has quit IRC | 20:47 | |
| *** thors____ has joined #openstack-infra | 20:47 | |
| *** thorst__ has quit IRC | 20:47 | |
| *** thorst__ has joined #openstack-infra | 20:48 | |
| notmyname | mordred: clarkb: FWIW, the rax team has a jenkins instance doing unit and functional tests against the golang branch http://104.239.166.47/job/Swift%20Golang%20Checks/163/console | 20:48 |
| fungi | notmyname: ooh, that's a great point of reference to crib from (though would be better with set -x) | 20:49 |
| *** mixos has joined #openstack-infra | 20:49 | |
| notmyname | fungi: let's consider all of this as POC and examples :-) | 20:49 |
| *** thorst___ has quit IRC | 20:49 | |
| *** jsavak has quit IRC | 20:49 | |
| fungi | notmyname: yeah, just meant i'm uncertain all the commands run are displayed in that log, so harder to crib from ;) | 20:49 |
| notmyname | fungi: I added a link to it on https://etherpad.openstack.org/p/golang-infra-issues-to-solve | 20:50 |
| *** yamahata has joined #openstack-infra | 20:50 | |
| notmyname | fungi: the interesting part for me is the unit test running at the top | 20:50 |
| *** thorst_ has quit IRC | 20:50 | |
| timothyb89 | krotscheck, mtreinish: on the topic of npm performance, have you seen https://github.com/alexanderGugel/ied ? I've seen dramatic speedups for running 'npm install', ~5m -> 30s | 20:50 |
| mordred | notmyname: cool | 20:50 |
| openstackgerrit | Hongbin Lu proposed openstack/requirements: Add Higgins project to projects.txt https://review.openstack.org/315249 | 20:50 |
| notmyname | fungi: ie it makes a lot of sense to "build deploy test" when you're talking about functional tests, but the golang unittest stuff is new to me | 20:51 |
| mordred | WOW | 20:51 |
| mordred | ERROR: ld.so: object '/var/lib/jenkins/.java/jenkins.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. | 20:51 |
| notmyname | and the overal good thing is showing both python and golang code in the same project being tests | 20:51 |
| fungi | notmyname: if you can tease a full job definition out of them, we might not have to guess at as much stuff | 20:51 |
| *** thors____ has quit IRC | 20:51 | |
| mordred | I do not want jenkins.so to be LD_PRELOADED | 20:51 |
| mordred | fungi: what fungi said | 20:51 |
| mordred | gah | 20:51 |
| fungi | mordred: thankfully it's not preloaded | 20:51 |
| mordred | notmyname: what fungi said - I'd love to know what commands it's running | 20:51 |
| notmyname | mordred: again, don't worry so much about this particular thing and if it's exactly how you'd want it run :-) | 20:52 |
| mordred | like - is that a script running go get ; go vet ; go test ? | 20:52 |
| mordred | or the make file | 20:52 |
| *** thors____ has joined #openstack-infra | 20:52 | |
| notmyname | mordred: fungi: I'll ask/investigate | 20:52 |
| mordred | notmyname: totally - mostly just curious as to what's happening now | 20:52 |
| fungi | notmyname: awesome--thanks! | 20:52 |
| *** thorst__ has quit IRC | 20:52 | |
| fungi | yeah, similar to mordred i mostly want to know what commands their job scripts are running, since that provides a much more solid starting point | 20:52 |
| fungi | not that we would necessarily do exactly the same thing | 20:53 |
| *** thorst__ has joined #openstack-infra | 20:53 | |
| *** thor_____ has joined #openstack-infra | 20:54 | |
| clarkb | mordred I always ld preload jenkins for maximum CI | 20:54 |
| mordred | clarkb: you ldpreload everytihng though | 20:54 |
| *** thorst___ has joined #openstack-infra | 20:55 | |
| *** thorst_ has joined #openstack-infra | 20:56 | |
| *** thors____ has quit IRC | 20:56 | |
| pabelanger | mordred: did you want to do something on nodepool.o.o were ready to do restarts | 20:56 |
| *** xyang has joined #openstack-infra | 20:56 | |
| mordred | pabelanger: ooh! yeah. did you land your patches? | 20:57 |
| *** thors____ has joined #openstack-infra | 20:57 | |
| pabelanger | mordred: indeed, they are on nodepool.o.o now | 20:57 |
| openstackgerrit | Merged openstack-infra/shade: Amend the valid fields to update on recordsets https://review.openstack.org/313520 | 20:57 |
| mordred | woot. SO ... | 20:57 |
| *** thorst__ has quit IRC | 20:57 | |
| mordred | fungi, clarkb, pabelanger: https://review.openstack.org/#/c/307755/ - I'd love to apply this too, if y'all are ok with that. | 20:58 |
| mordred | but we can also let pabelanger's changes land today and do that tomorrow | 20:58 |
| openstackgerrit | Merged openstack-infra/glean: Enable support for configuring bonded interfaces https://review.openstack.org/293648 | 20:58 |
| *** thorst__ has joined #openstack-infra | 20:58 | |
| *** jamesmca_ has joined #openstack-infra | 20:58 | |
| *** thor_____ has quit IRC | 20:58 | |
| *** weshay has quit IRC | 20:59 | |
| pabelanger | I'm fine landing it today | 20:59 |
| *** thorst___ has quit IRC | 20:59 | |
| *** xyang has quit IRC | 20:59 | |
| *** thorst_ has quit IRC | 21:00 | |
| *** mjblack has quit IRC | 21:01 | |
| *** aratus has joined #openstack-infra | 21:02 | |
| *** thors____ has quit IRC | 21:02 | |
| openstackgerrit | Tim Buckley proposed openstack-infra/project-config: Create stackviz element and script in nodepool https://review.openstack.org/311815 | 21:02 |
| *** davideagnello has joined #openstack-infra | 21:02 | |
| *** thorst__ has quit IRC | 21:02 | |
| cloudnull | clarkb: i'm not seeing any smoking gun regarding vm build + floatingIP assignment. | 21:05 |
| *** xyang has joined #openstack-infra | 21:05 | |
| *** sridhar_ram1 has quit IRC | 21:05 | |
| cloudnull | messaging had a fit on the 8 which caused some traces in the nova logs, but nothing notable since then | 21:05 |
| *** abregman has joined #openstack-infra | 21:05 | |
| openstackgerrit | Tim Buckley proposed openstack-infra/project-config: Create stackviz element and script in nodepool https://review.openstack.org/311815 | 21:06 |
| cloudnull | neutron seems to be happy at this time . | 21:06 |
| *** mjblack has joined #openstack-infra | 21:06 | |
| *** aysyd has quit IRC | 21:06 | |
| *** hashar_ has joined #openstack-infra | 21:06 | |
| mtreinish | timothyb89: I haven't tried that before. I'll give it a look | 21:07 |
| *** baumann has left #openstack-infra | 21:07 | |
| pabelanger | mordred: clarkb: I'm going to step out for a moment to get some food. Everything is in place for the new nodepool-builder worker flags. I should be back shortly | 21:07 |
| *** eharney has quit IRC | 21:08 | |
| cloudnull | Im going to fire off a bunch of builds/assosiates to see if i can recreate it and trap something from the logs. | 21:08 |
| *** hrybacki has quit IRC | 21:09 | |
| *** tonytan4ever has quit IRC | 21:10 | |
| *** hrybacki has joined #openstack-infra | 21:10 | |
| *** thorst_ has joined #openstack-infra | 21:10 | |
| *** psilvad has quit IRC | 21:10 | |
| *** hashar has quit IRC | 21:11 | |
| *** ekarlso has quit IRC | 21:11 | |
| *** thorst__ has joined #openstack-infra | 21:11 | |
| *** abregman has quit IRC | 21:12 | |
| *** thorst___ has joined #openstack-infra | 21:12 | |
| *** thors____ has joined #openstack-infra | 21:14 | |
| *** davideagnello has quit IRC | 21:14 | |
| *** ekarlso has joined #openstack-infra | 21:14 | |
| *** thor_____ has joined #openstack-infra | 21:15 | |
| *** thorst_ has quit IRC | 21:15 | |
| *** ayoung has joined #openstack-infra | 21:15 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| *** davideagnello has joined #openstack-infra | 21:16 | |
| *** thorst__ has quit IRC | 21:16 | |
| *** thorst_ has joined #openstack-infra | 21:16 | |
| *** IlyaG has joined #openstack-infra | 21:17 | |
| *** jamesmca_ has quit IRC | 21:17 | |
| *** thorst___ has quit IRC | 21:17 | |
| *** thorst__ has joined #openstack-infra | 21:18 | |
| openstackgerrit | Merged openstack/requirements: Update oslo.versonedobjects to 1.9.1 and block 1.9.0 https://review.openstack.org/315164 | 21:18 |
| *** thors____ has quit IRC | 21:18 | |
| *** thorst___ has joined #openstack-infra | 21:19 | |
| *** thor_____ has quit IRC | 21:19 | |
| *** thors____ has joined #openstack-infra | 21:20 | |
| openstackgerrit | Merged openstack/requirements: Remove kolla-mesos from projects.txt https://review.openstack.org/313262 | 21:20 |
| *** vilobhmm11 has joined #openstack-infra | 21:20 | |
| *** matrohon has quit IRC | 21:20 | |
| *** thor_____ has joined #openstack-infra | 21:21 | |
| *** thorst_ has quit IRC | 21:21 | |
| *** keedya has quit IRC | 21:22 | |
| *** thorst_ has joined #openstack-infra | 21:22 | |
| clarkb | cloudnull: ok when I am not out running errands I can check our logs | 21:22 |
| *** thorst__ has quit IRC | 21:22 | |
| *** thorst__ has joined #openstack-infra | 21:23 | |
| *** thorst___ has quit IRC | 21:23 | |
| *** jordanP has quit IRC | 21:23 | |
| *** thors____ has quit IRC | 21:24 | |
| *** xyang has left #openstack-infra | 21:24 | |
| *** thorst___ has joined #openstack-infra | 21:24 | |
| *** rhallisey has quit IRC | 21:24 | |
| *** thor_____ has quit IRC | 21:25 | |
| *** thors____ has joined #openstack-infra | 21:25 | |
| *** regXboi has quit IRC | 21:25 | |
| *** thor_____ has joined #openstack-infra | 21:26 | |
| *** thorst_ has quit IRC | 21:26 | |
| *** lakshmiS_ has quit IRC | 21:26 | |
| *** |-paul-| has joined #openstack-infra | 21:27 | |
| *** thorst_ has joined #openstack-infra | 21:27 | |
| *** thorst__ has quit IRC | 21:27 | |
| *** hashar_ has quit IRC | 21:27 | |
| *** thorst__ has joined #openstack-infra | 21:28 | |
| *** thorst___ has quit IRC | 21:28 | |
| *** thorst___ has joined #openstack-infra | 21:29 | |
| *** zz_dimtruck is now known as dimtruck | 21:29 | |
| *** thors____ has quit IRC | 21:29 | |
| *** nadya has quit IRC | 21:30 | |
| *** esikache1 has quit IRC | 21:30 | |
| *** thors____ has joined #openstack-infra | 21:30 | |
| *** thor_____ has quit IRC | 21:30 | |
| *** jamesmca_ has joined #openstack-infra | 21:30 | |
| *** baoli has quit IRC | 21:31 | |
| *** thorst_ has quit IRC | 21:31 | |
| *** thorst_ has joined #openstack-infra | 21:31 | |
| *** thor_____ has joined #openstack-infra | 21:32 | |
| *** thorst__ has quit IRC | 21:32 | |
| *** vilobhmm11 has left #openstack-infra | 21:33 | |
| *** thorst___ has quit IRC | 21:33 | |
| *** thorst__ has joined #openstack-infra | 21:33 | |
| *** thorst___ has joined #openstack-infra | 21:34 | |
| *** edmondsw has quit IRC | 21:34 | |
| *** thors____ has quit IRC | 21:35 | |
| *** thorst_ has quit IRC | 21:35 | |
| *** thors____ has joined #openstack-infra | 21:35 | |
| openstackgerrit | Dirk Mueller proposed openstack/requirements: Clarify wording by mentioning the name of the gate job https://review.openstack.org/315262 | 21:35 |
| *** thorst_ has joined #openstack-infra | 21:36 | |
| *** thor_____ has quit IRC | 21:36 | |
| *** thor_____ has joined #openstack-infra | 21:37 | |
| *** thorst__ has quit IRC | 21:38 | |
| *** Sukhdev has quit IRC | 21:38 | |
| *** thorst__ has joined #openstack-infra | 21:39 | |
| *** thorst___ has quit IRC | 21:39 | |
| *** savihou has quit IRC | 21:39 | |
| *** thorst___ has joined #openstack-infra | 21:40 | |
| *** thors____ has quit IRC | 21:40 | |
| *** thorst_ has quit IRC | 21:41 | |
| *** thors____ has joined #openstack-infra | 21:41 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add oslo.tools repo/project https://review.openstack.org/314342 | 21:41 |
| *** jang has quit IRC | 21:41 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config: Enable translation jobs for manila-ui https://review.openstack.org/314905 | 21:41 |
| *** thor_____ has quit IRC | 21:42 | |
| *** thorst_ has joined #openstack-infra | 21:42 | |
| *** thorst__ has quit IRC | 21:43 | |
| *** thorst__ has joined #openstack-infra | 21:44 | |
| *** thorst___ has quit IRC | 21:44 | |
| *** thors____ has quit IRC | 21:45 | |
| *** thorst___ has joined #openstack-infra | 21:45 | |
| *** ddieterly is now known as ddieterly[away] | 21:45 | |
| *** kzaitsev_mb has quit IRC | 21:45 | |
| *** thors____ has joined #openstack-infra | 21:46 | |
| *** mriedem has quit IRC | 21:47 | |
| *** jamesmca_ has quit IRC | 21:47 | |
| *** julim has quit IRC | 21:47 | |
| *** pradk has quit IRC | 21:47 | |
| *** tho______ has joined #openstack-infra | 21:47 | |
| *** gordc has quit IRC | 21:47 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Fix apt-sources configuration for debian-minimal https://review.openstack.org/315265 | 21:47 |
| *** thorst_ has quit IRC | 21:47 | |
| *** thorst__ has quit IRC | 21:48 | |
| *** thorst_ has joined #openstack-infra | 21:48 | |
| *** ddieterly[away] is now known as ddieterly | 21:48 | |
| *** HeOS has joined #openstack-infra | 21:48 | |
| *** Goneri has quit IRC | 21:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:48 | |
| *** thorst__ has joined #openstack-infra | 21:49 | |
| *** thorst___ has quit IRC | 21:49 | |
| *** claudiub has quit IRC | 21:50 | |
| openstackgerrit | Dirk Mueller proposed openstack/requirements: Removes posix-ipc from global-requirements https://review.openstack.org/217824 | 21:50 |
| *** thors____ has quit IRC | 21:50 | |
| kevinbenton | any gerrty devs around? | 21:51 |
| *** tho______ has quit IRC | 21:52 | |
| pabelanger | mordred: clarkb: okay, I'm going to restart nodepool now minus 307755. | 21:52 |
| *** jamesmca_ has joined #openstack-infra | 21:52 | |
| *** thorst_ has quit IRC | 21:53 | |
| *** thorst__ has quit IRC | 21:53 | |
| *** jsavak has joined #openstack-infra | 21:55 | |
| *** HeOS has quit IRC | 21:55 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project https://review.openstack.org/315267 | 21:56 |
| pabelanger | #status log nodepool restarted to pickup https://review.openstack.org/#/c/294339/ | 21:56 |
| *** mkwiek has quit IRC | 21:56 | |
| openstackstatus | pabelanger: finished logging | 21:56 |
| *** Swami has quit IRC | 21:57 | |
| *** mkwiek has joined #openstack-infra | 21:58 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project https://review.openstack.org/315267 | 21:58 |
| *** austin81 has left #openstack-infra | 21:59 | |
| *** njohnston is now known as njohnston_afk | 22:00 | |
| *** njohnston_afk is now known as njohnston__ | 22:00 | |
| *** thorst_ has joined #openstack-infra | 22:01 | |
| *** xyang1 has quit IRC | 22:01 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project https://review.openstack.org/315267 | 22:01 |
| *** rcrit has quit IRC | 22:02 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin https://review.openstack.org/314862 | 22:04 |
| hongbin | Hi, I have submitted a request to create a project: https://review.openstack.org/#/c/313935/ . I wonder if anyone can add me to these two gerrit groups: higgins-core, higgins-release | 22:04 |
| *** ddieterly is now known as ddieterly[away] | 22:04 | |
| *** kien-ha has quit IRC | 22:05 | |
| *** javeriak_ has quit IRC | 22:05 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add extra metadata to the tag messages https://review.openstack.org/314141 | 22:05 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix previous version detection for repos with no tags https://review.openstack.org/314234 | 22:05 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: automatically submit constraint updates after tagging https://review.openstack.org/314111 | 22:05 |
| *** jamesmca_ has quit IRC | 22:06 | |
| *** sridhar_ram has joined #openstack-infra | 22:07 | |
| pabelanger | jeblair: fungi: clarkb: do you have a moment to help with nodepool.o.o, I am not sure things are coming back online as expected | 22:08 |
| clarkb | I am about 5 minutes from home can get on then | 22:09 |
| pabelanger | clarkb: great, thanks | 22:09 |
| *** annegentle has quit IRC | 22:10 | |
| *** cloudtrainme has joined #openstack-infra | 22:10 | |
| *** jsavak has quit IRC | 22:12 | |
| clarkb | ok here now | 22:12 |
| *** jordanP has joined #openstack-infra | 22:13 | |
| pabelanger | clarkb: sure, so if you look at nodepool.log, it appear nodepoold is no longer doing anything | 22:13 |
| pabelanger | clarkb: it tried to mass delete some nodes but that's the last traffic | 22:13 |
| *** thorst_ has quit IRC | 22:14 | |
| *** jordanP has quit IRC | 22:14 | |
| clarkb | pabelanger: we have a thread dump signal handler, I think it is sigusr2 | 22:15 |
| clarkb | let me double check | 22:15 |
| clarkb | we can run that to try an dsee what is going on | 22:15 |
| pabelanger | right, I forgot about that | 22:15 |
| clarkb | ya sigusr2 so if you kill -USR2 $pid you should get that all in the debug log | 22:16 |
| pabelanger | sure, 1 sec | 22:16 |
| *** dimtruck is now known as zz_dimtruck | 22:16 | |
| *** davideagnello has quit IRC | 22:17 | |
| *** mixos has quit IRC | 22:17 | |
| pabelanger | ran command | 22:17 |
| pabelanger | nothing? | 22:18 |
| pabelanger | clarkb: Ya, nothing in debug.log | 22:18 |
| clarkb | against 6862 ? | 22:18 |
| pabelanger | yup | 22:18 |
| clarkb | if so then I think its probably really hosed | 22:18 |
| pabelanger | okay, how should we proceed? | 22:19 |
| clarkb | which changes did we add in? | 22:19 |
| clarkb | was it just the builder worker change? that shouldn't affect nodepoold | 22:19 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:19 | |
| pabelanger | ya just 294339 | 22:19 |
| clarkb | I have a strace against it and its doing some things | 22:20 |
| clarkb | looks like zmq socket is still alive and receiving things | 22:20 |
| openstackgerrit | Itisha Dewan proposed openstack-infra/project-config: Adding library Delimiter to the CI system https://review.openstack.org/315232 | 22:20 |
| *** thorst_ has joined #openstack-infra | 22:20 | |
| clarkb | pabelanger: we can try restarting it, my only fear with that is since the only change to nodepool itself should be indepdnent of this daemon that the issue is in a dep somewhere like sahde or paramiko or whatever | 22:21 |
| pabelanger | clarkb: maybe we pulled in a newer version of shade | 22:21 |
| pabelanger | shade==1.8.0 is installed | 22:22 |
| pabelanger | not sure what we had before | 22:22 |
| clarkb | ya I thought we were already running with that based on when it released and when jeblair restarted for the zuul 2.5 things | 22:22 |
| clarkb | anyways with no stack dumping I think only thing we really can do is restart and try to get better info if there is a next time | 22:23 |
| pabelanger | we also have paramiko==2.0.0 | 22:23 |
| clarkb | its almost like the main thread died but we have a non daemon thread running keeping the process alive | 22:23 |
| pabelanger | clarkb: okay, did you want to restart it? | 22:23 |
| clarkb | pabelanger: you can do it, just stop then start | 22:23 |
| clarkb | if the main thread did die we might be able to see a traceback in the logs | 22:24 |
| pabelanger | okay stopped | 22:24 |
| pabelanger | and starting | 22:24 |
| *** thorst_ has quit IRC | 22:25 | |
| clarkb | it does while True: signal.pause() in the main thread | 22:25 |
| JayF | where are the logs for post jobs kept? | 22:25 |
| clarkb | so even if it gets signals it should go back to pausing, only thing would be an exception I Think | 22:26 |
| clarkb | but there is no logging wrapped aroun dthat :/ | 22:26 |
| clarkb | pabelanger: it seems stuck again | 22:26 |
| pabelanger | clarkb: yup | 22:26 |
| JayF | nevermind, jroll beat me to the punch | 22:26 |
| clarkb | JayF: http://logs.openstack.org/first2ofsha1/sha1 | 22:26 |
| *** yamamoto has quit IRC | 22:28 | |
| *** yamamoto has joined #openstack-infra | 22:28 | |
| pabelanger | kill -USR2 not working still | 22:28 |
| *** zz_dimtruck has quit IRC | 22:28 | |
| clarkb | pabelanger: hrm we did update nodepoold for the workers change but its all hidden behind the if args.builder check | 22:28 |
| *** dprince has quit IRC | 22:29 | |
| *** Goneri has joined #openstack-infra | 22:29 | |
| pabelanger | clarkb: ya, pretty sure I tested it locally too | 22:30 |
| pabelanger | we can try rolling back to see if that helps | 22:30 |
| clarkb | and the tests would test it as well | 22:30 |
| clarkb | I don't think it is the problem | 22:30 |
| pabelanger | okay | 22:30 |
| clarkb | I keep coming back to main thread is crashing and we have a non daemon thread keeping the process running somewhere | 22:31 |
| clarkb | we could try a ninja try: except: around that signal.pause and log stuff | 22:31 |
| clarkb | or maybe it is crashing before that | 22:31 |
| pabelanger | what about nodepoold -d ? | 22:31 |
| *** zz_dimtruck has joined #openstack-infra | 22:31 | |
| *** zz_dimtruck is now known as dimtruck | 22:31 | |
| *** sridhar_ram1 has joined #openstack-infra | 22:32 | |
| *** HeOS has joined #openstack-infra | 22:32 | |
| greghaynes | I really don't see how that change would cause nodepoold to hang | 22:32 |
| *** sridhar_ram1 has quit IRC | 22:32 | |
| *** ddieterly[away] is now known as ddieterly | 22:32 | |
| greghaynes | Are you sure there wasnt another change sitting on disk that hadnt been used due to no restart? | 22:33 |
| pabelanger | /opt/nodepool looks good | 22:33 |
| pabelanger | we install from there | 22:33 |
| *** sridhar_ram1 has joined #openstack-infra | 22:34 | |
| clarkb | greghaynes: not 100% sure but reasonably sure | 22:34 |
| *** roxanaghe has quit IRC | 22:34 | |
| *** sridhar_ram has quit IRC | 22:34 | |
| clarkb | greghaynes: jeblair restarted it during summit for zuul 2.5 stuff | 22:34 |
| clarkb | jeblair: ^ can you confirm | 22:34 |
| kevinbenton | jeblair: hey, do you have any thoughts on gertty tracking a pid to check for multiple instances talking to same DB? I've ruined my local gertty database several times by running multiple on accident | 22:35 |
| *** ddieterly has quit IRC | 22:36 | |
| greghaynes | Yea, 294339 just doesnt hit any nodepoold code paths | 22:36 |
| *** roxanaghe has joined #openstack-infra | 22:37 | |
| clarkb | pabelanger: I also doule checked that pbr/pip claim to have hte currently checked out version installed and they do | 22:37 |
| clarkb | so we aren't running some random broken code | 22:38 |
| clarkb | (I don't think) | 22:38 |
| pabelanger | clarkb: Ya, I was hoping we logged pip install some place, but don't see it | 22:38 |
| clarkb | pabelanger: its in syslog for puppet | 22:38 |
| *** thorst_ has joined #openstack-infra | 22:39 | |
| pabelanger | clarkb: really? I cannot see the output of pip install -U /opt/nodepool | 22:39 |
| clarkb | pabelanger: oh you won't get the full output of that there, that goes in ~root/.pip | 22:39 |
| clarkb | you do get it if it explodes I think | 22:39 |
| pabelanger | okay, checking that | 22:39 |
| *** rbrndt has quit IRC | 22:39 | |
| clarkb | hrm not ~root/.pip | 22:40 |
| pabelanger | ya | 22:40 |
| clarkb | does puppet not set $HOME properly? | 22:40 |
| pabelanger | not sure | 22:40 |
| *** thorst___ has joined #openstack-infra | 22:41 | |
| clarkb | wrapping all of main() in a signal handler and logging that is my next best idea | 22:41 |
| *** Swami has joined #openstack-infra | 22:41 | |
| clarkb | let me write that patch really quick | 22:41 |
| *** thors____ has joined #openstack-infra | 22:42 | |
| pabelanger | not that it help, but here are the last changes in /usr/local/lib/python2.7/dist-packages | 22:42 |
| greghaynes | Is it starting, not logging anything, and just hanging? | 22:42 |
| pabelanger | http://paste.openstack.org/show/496829/ | 22:42 |
| *** rbrndt has joined #openstack-infra | 22:42 | |
| clarkb | greghaynes: it is starting but most of the owkr happens in the pool thread | 22:43 |
| *** thor_____ has joined #openstack-infra | 22:43 | |
| clarkb | so it runs and then at some point stops doing things excet reading zmq events according to strace | 22:43 |
| clarkb | which is why I think its main thread dying and all daemon threads going iwth it | 22:43 |
| greghaynes | fun, I bet zmq is just a totally separate thread | 22:43 |
| clarkb | pabelanger: running with -d would be potentially anothre way to debug it | 22:43 |
| greghaynes | yea, or just blocking forever | 22:43 |
| *** thorst_ has quit IRC | 22:43 | |
| clarkb | greghaynes: ya that is other possibility | 22:43 |
| greghaynes | the next couple changes before this latest one deal with locks | 22:44 |
| pabelanger | clarkb: agreed | 22:44 |
| *** thorst_ has joined #openstack-infra | 22:44 | |
| *** thorst__ has joined #openstack-infra | 22:45 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Log when nodepoold main excepts and dies https://review.openstack.org/315278 | 22:45 |
| *** thorst___ has quit IRC | 22:45 | |
| clarkb | so thats one option ^ | 22:45 |
| clarkb | greghaynes: so maybe jebalir didn't restart it with those changes? or we are somehow able to tickle a deadlock now when we were not ble to before | 22:46 |
| greghaynes | LGTM | 22:46 |
| *** thorst___ has joined #openstack-infra | 22:46 | |
| greghaynes | clarkb: Yea, either of those things. I just really dont see it being caused by 294339 | 22:46 |
| *** thors____ has quit IRC | 22:47 | |
| clarkb | agreed | 22:47 |
| *** thors____ has joined #openstack-infra | 22:47 | |
| clarkb | pabelanger: so thats another option, do a wholesale revert install | 22:47 |
| *** thor_____ has quit IRC | 22:47 | |
| *** tpsilva has quit IRC | 22:47 | |
| *** dmk0202 has quit IRC | 22:47 | |
| fungi | pabelanger: i'm back around now, sorry... was busy making dinner | 22:48 |
| clarkb | 7c9c0516c54097159e1307fc5b5b48467d8faa8b maybe | 22:48 |
| pabelanger | clarkb: I'm find with reverting to confirm the issue is not related to 294339 | 22:48 |
| *** thor_____ has joined #openstack-infra | 22:48 | |
| clarkb | use ^ that commit as the checkout to install | 22:48 |
| *** thorst_ has quit IRC | 22:48 | |
| clarkb | greghaynes: ^ is that one look right to you based on commit diffs? | 22:48 |
| *** thorst_ has joined #openstack-infra | 22:49 | |
| *** thorst__ has quit IRC | 22:49 | |
| pabelanger | clarkb: sure, /opt/nodepool or local clone | 22:49 |
| pabelanger | not sure what people do when we revert | 22:49 |
| clarkb | pabelanger: /opt/nodepool, just disable puppet | 22:49 |
| greghaynes | clarkb: as a safe starting spot? possibly, need to make sure that the changes in front are ok to back out on... | 22:49 |
| clarkb | pabelanger: my usual practice is stop puppet, git checkout manually, pip install manually | 22:50 |
| clarkb | then based on that we can reflect it in tree and reenable puppet | 22:50 |
| *** thorst__ has joined #openstack-infra | 22:50 | |
| pabelanger | clarkb: okay, added nodepool.o.o to emergency file | 22:50 |
| clarkb | greghaynes: the only thing I am worried about is we added a column | 22:50 |
| clarkb | greghaynes: but that column should be fine as just being in the db unused I think | 22:50 |
| *** |-paul-| has quit IRC | 22:50 | |
| *** HeOS has quit IRC | 22:50 | |
| greghaynes | eek | 22:50 |
| *** thorst___ has quit IRC | 22:51 | |
| jeblair | we were running e0f before the latest restart | 22:51 |
| fungi | clarkb: pabelanger: what if it's trying to reach the tripleo cloud? tried commenting that out of the config? | 22:51 |
| *** thorst___ has joined #openstack-infra | 22:51 | |
| pabelanger | fungi: Oh, that is interesting. Since it is down | 22:51 |
| clarkb | fungi: I don't think it is, at least if that was the problem sigusr2 should function | 22:51 |
| fungi | still digesting what's been tried and the current symptoms, so ignore me if i'm in left field on that | 22:51 |
| clarkb | since we install the signal handler before we start the pool | 22:52 |
| *** thors____ has quit IRC | 22:52 | |
| clarkb | jeblair: ok thanks for confirming | 22:52 |
| fungi | but yeah, just keep in mind that one of our providers is down hard and unreachable right now | 22:52 |
| *** thors____ has joined #openstack-infra | 22:52 | |
| pabelanger | that should be easy to test | 22:53 |
| *** thor_____ has quit IRC | 22:53 | |
| *** thorst_ has quit IRC | 22:53 | |
| clarkb | ya just remove it from the config temporarily | 22:53 |
| greghaynes | If that doesnt do the trick, sounds like next best option is to try clarkb's debug patch | 22:54 |
| jeblair | you can also run it in the foreground | 22:54 |
| *** thorst_ has joined #openstack-infra | 22:54 | |
| pabelanger | clarkb: remove or min-ready: -1? | 22:54 |
| *** thorst__ has quit IRC | 22:54 | |
| fungi | suddenly glad we locked down our jenkins masters and are working to excise them entirely... http://www.openwall.com/lists/oss-security/2016/05/11/15 | 22:54 |
| *** thorst___ has quit IRC | 22:55 | |
| pleia2 | wow, that's worse than I thought :\ | 22:56 |
| clarkb | pabelanger: you have to completely ocmment it out if you don't want it talking to it at all | 22:56 |
| jeblair | the fist few are features, the last few are :( | 22:56 |
| *** thorst___ has joined #openstack-infra | 22:56 | |
| clarkb | pabelanger: -1 will still make clients and some connections just not do any real work | 22:56 |
| *** thors____ has quit IRC | 22:56 | |
| pabelanger | clarkb: okay, removed | 22:56 |
| pabelanger | stopping nodepool | 22:56 |
| clarkb | pabelanger: I think if that doens't fix it then we try running it in foreground | 22:56 |
| *** HeOS has joined #openstack-infra | 22:57 | |
| pabelanger | clarkb: sure | 22:57 |
| clarkb | and if that doesn't help then we move on to the more drastic measures of debug patch or reverts | 22:57 |
| pabelanger | and starting now | 22:57 |
| pabelanger | running again | 22:57 |
| *** thors____ has joined #openstack-infra | 22:57 | |
| *** thor_____ has joined #openstack-infra | 22:58 | |
| *** thorst_ has quit IRC | 22:58 | |
| *** Jeffrey4l has joined #openstack-infra | 22:59 | |
| *** thorst_ has joined #openstack-infra | 22:59 | |
| fungi | aside from tracebacks about the unrecognized images from tripleo, the debug log looks sane | 22:59 |
| pabelanger | yup | 22:59 |
| greghaynes | oh, that did it? | 22:59 |
| pabelanger | not sure, but still running | 22:59 |
| *** kzaitsev_mb has quit IRC | 23:00 | |
| *** thorst__ has joined #openstack-infra | 23:00 | |
| clarkb | that is really weird if so, sigusr2 is installed before pool is started | 23:00 |
| * fungi hates being right about this sort of thing | 23:00 | |
| clarkb | maybe we are doing more than we should on object instantiation | 23:00 |
| *** thorst___ has quit IRC | 23:00 | |
| pabelanger | fungi: I'll be happy if you are right! | 23:00 |
| greghaynes | Yes, looks like fungi is the winner? | 23:00 |
| *** thorst___ has joined #openstack-infra | 23:01 | |
| fungi | yeah, debug log is pretty unhappy trying to delete some nodes too, all in tripleo cloud i think | 23:01 |
| clarkb | this should be easy to test with the devstack plugin | 23:01 |
| fungi | but that should be benign just noisy | 23:01 |
| clarkb | make cloud, turn off cloud, restart nodepool | 23:01 |
| fungi | can call it "the tripleo cloud maintenance test" | 23:01 |
| clarkb | we can likely just remove all those db roews since I doubt the new hardware will continue to host those | 23:01 |
| *** thors____ has quit IRC | 23:01 | |
| fungi | agreed | 23:01 |
| *** HeOS has quit IRC | 23:01 | |
| *** rbrndt has quit IRC | 23:01 | |
| *** jamielennox is now known as jamielennox|away | 23:02 | |
| *** thors____ has joined #openstack-infra | 23:02 | |
| pabelanger | I haven't noticed any new nodes coming online yet however | 23:03 |
| *** thor_____ has quit IRC | 23:03 | |
| clarkb | that may just be nova boot lag | 23:03 |
| pabelanger | but nodepool is still working | 23:03 |
| *** thor_____ has joined #openstack-infra | 23:03 | |
| *** thorst_ has quit IRC | 23:04 | |
| greghaynes | That is really crazy that we can block the main thread if a cloud is down... that implies we are somehow connecting via the main thread | 23:04 |
| *** thorst__ has quit IRC | 23:04 | |
| *** thorst_ has joined #openstack-infra | 23:04 | |
| pabelanger | Ya, it appears that way | 23:05 |
| *** hongbin has quit IRC | 23:05 | |
| *** thorst__ has joined #openstack-infra | 23:05 | |
| clarkb | its possible the setup for pool which happens before signal handlers is at fault for that | 23:05 |
| clarkb | if we do too much in setup | 23:05 |
| fungi | the only reason i even thought to mention it is that we've seen this behavior before. i thought it was fixed, but now i don't remember what the fix was supposed to be | 23:05 |
| *** thorst___ has quit IRC | 23:05 | |
| clarkb | we should do setup logging then setup signal handlers, then everything else | 23:06 |
| *** thorst___ has joined #openstack-infra | 23:06 | |
| greghaynes | clarkb: set up? AFAICT its just the constructor for NodePool which is called and that only makes threads and does the statsd connection badness | 23:06 |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Oslo releases for week of may 9th https://review.openstack.org/314338 | 23:06 |
| greghaynes | er, not makes threads, makes thread conditions | 23:07 |
| clarkb | greghaynes: ya the threads all connect to clouds | 23:07 |
| *** thors____ has quit IRC | 23:07 | |
| clarkb | but let me double check | 23:07 |
| *** thor_____ has quit IRC | 23:08 | |
| *** r-daneel has quit IRC | 23:08 | |
| *** thors____ has joined #openstack-infra | 23:08 | |
| clarkb | ah yup, it basically does nothing | 23:08 |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Bump uc for oslo.service 1.10.0 https://review.openstack.org/314307 | 23:08 |
| greghaynes | yea, everything is as a result of run() | 23:08 |
| clarkb | now I am really confused, maybe python GIL isn't being released to let main() handle signal? | 23:08 |
| greghaynes | oh, you know | 23:09 |
| clarkb | go down on a read or connect to cloud, block indepfinitely, never hand over thread control | 23:09 |
| greghaynes | hrm, actually, no, I was thinking maybe it was blocking on dns resolution which might mess up the GIL | 23:09 |
| *** thorst_ has quit IRC | 23:09 | |
| greghaynes | but dns should be fine right now | 23:09 |
| *** thor_____ has joined #openstack-infra | 23:09 | |
| *** thorst__ has quit IRC | 23:09 | |
| clarkb | ya we have dns servers that should nxdomain if tripleo deleted records | 23:09 |
| *** kmartin has joined #openstack-infra | 23:10 | |
| *** yuanying has joined #openstack-infra | 23:10 | |
| *** thorst___ has quit IRC | 23:10 | |
| *** thorst__ has joined #openstack-infra | 23:11 | |
| *** IlyaG has quit IRC | 23:11 | |
| clarkb | in any case we should be able to reproduce easily enough using the devstakc plugin, I won't get to that now or today though | 23:12 |
| greghaynes | yep | 23:12 |
| *** thorst___ has joined #openstack-infra | 23:12 | |
| *** sdague has quit IRC | 23:12 | |
| pabelanger | clarkb: do we need to manually purge the database now? | 23:12 |
| *** gildub has joined #openstack-infra | 23:12 | |
| *** sambetts|afk has quit IRC | 23:13 | |
| *** SergK has quit IRC | 23:13 | |
| clarkb | pabelanger: if tripleo doesn't preserve those instances and images, yes | 23:13 |
| pabelanger | or will nodepool recover the deleted nodes eventually (minus tripleo) | 23:13 |
| fungi | pabelanger: yeah, just delete from various tables based on %tripleo% pattern | 23:13 |
| *** thors____ has quit IRC | 23:13 | |
| *** thors____ has joined #openstack-infra | 23:13 | |
| jhesketh | Morning | 23:13 |
| clarkb | the non tripleo stuff should be cleaned up automagically | 23:13 |
| pabelanger | still not sure why nodepool is not building any nodes yet | 23:13 |
| *** thor_____ has quit IRC | 23:13 | |
| fungi | i don't think nodepool is going to clean this up until the tripleo cloud comes back into service otherwise | 23:13 |
| clarkb | oh ya it may be dying on the tripleo ones | 23:13 |
| *** SergK has joined #openstack-infra | 23:13 | |
| pabelanger | that's what I am thinking too | 23:14 |
| *** kambiz has quit IRC | 23:14 | |
| *** zigo has quit IRC | 23:14 | |
| *** thor_____ has joined #openstack-infra | 23:14 | |
| pabelanger | I'm not to confident to do SQL queries on nodepool.o.o, any other takers? | 23:14 |
| *** pblaho has quit IRC | 23:14 | |
| *** YorikSar has quit IRC | 23:15 | |
| *** mikal has quit IRC | 23:15 | |
| *** Daviey has quit IRC | 23:15 | |
| *** pblaho has joined #openstack-infra | 23:15 | |
| *** mikal has joined #openstack-infra | 23:15 | |
| clarkb | I can walk you through it | 23:15 |
| pabelanger | sure | 23:15 |
| *** Daviey has joined #openstack-infra | 23:15 | |
| clarkb | let me get a sql connection going so I am not blind | 23:15 |
| *** zigo has joined #openstack-infra | 23:15 | |
| *** sambetts has joined #openstack-infra | 23:15 | |
| *** cburgess has quit IRC | 23:15 | |
| *** thorst_ has joined #openstack-infra | 23:15 | |
| *** YorikSar has joined #openstack-infra | 23:15 | |
| *** cburgess has joined #openstack-infra | 23:16 | |
| *** thorst__ has quit IRC | 23:16 | |
| *** thorst___ has quit IRC | 23:16 | |
| *** thorst__ has joined #openstack-infra | 23:17 | |
| pabelanger | so far I have select * from node where provider_name LIKE 'trip%'; | 23:17 |
| clarkb | pabelanger: the trick for deleting is you can do a select * with your where first, then if that listing looks good s/select */delete/ | 23:17 |
| *** kambiz has joined #openstack-infra | 23:17 | |
| clarkb | so select * from node where provider_name = 'tripleo-test-cloud-rh1'; | 23:17 |
| *** thors____ has quit IRC | 23:17 | |
| *** thorst___ has joined #openstack-infra | 23:18 | |
| clarkb | I would avoid like just because we don't have to match multiple values | 23:18 |
| pabelanger | okay, have the select working | 23:18 |
| *** thor_____ has quit IRC | 23:18 | |
| clarkb | pabelanger: if those records look good just replace the select * with delete | 23:19 |
| *** vhosakot has quit IRC | 23:19 | |
| clarkb | and that will remove those same rows | 23:19 |
| pabelanger | okay: running | 23:19 |
| pabelanger | delete from node where provider_name = 'tripleo-test-cloud-rh1'; | 23:19 |
| *** thors____ has joined #openstack-infra | 23:19 | |
| pabelanger | Query OK, 54 rows affected (0.02 sec) | 23:19 |
| pabelanger | confirmed removed | 23:19 |
| *** thorst_ has quit IRC | 23:19 | |
| clarkb | pabelanger: then select * from snapshot_image where provider_name = 'tripleo-test-cloud-rh1'; | 23:20 |
| pabelanger | okay, here we go | 23:20 |
| clarkb | and if that looks good reaplce select * with delete | 23:20 |
| *** thorst_ has joined #openstack-infra | 23:20 | |
| pabelanger | Query OK, 4 rows affected (0.00 sec) | 23:20 |
| pabelanger | great | 23:21 |
| clarkb | and thats it. we don't do multinode in tripleo so nothing in subnodes tables and dib images are provider independent | 23:21 |
| *** thorst__ has quit IRC | 23:21 | |
| pabelanger | ya | 23:21 |
| pabelanger | so, looks like nodepool won't build new nodes if it can't delete unknown cloud nodes | 23:22 |
| *** thorst__ has joined #openstack-infra | 23:22 | |
| *** thorst___ has quit IRC | 23:22 | |
| *** thorst___ has joined #openstack-infra | 23:23 | |
| *** thors____ has quit IRC | 23:23 | |
| *** thor_____ has joined #openstack-infra | 23:24 | |
| *** wajdi has quit IRC | 23:24 | |
| *** roxanaghe has quit IRC | 23:24 | |
| *** thorst_ has quit IRC | 23:24 | |
| pabelanger | oh, no. it is blocked again? | 23:24 |
| pabelanger | or just lag for me | 23:24 |
| *** thorst_ has joined #openstack-infra | 23:25 | |
| *** Liuqing has joined #openstack-infra | 23:25 | |
| pabelanger | clarkb: fungi: think the problem is back :( | 23:25 |
| *** thors____ has joined #openstack-infra | 23:26 | |
| *** thorst__ has quit IRC | 23:26 | |
| clarkb | woo | 23:26 |
| *** thorst__ has joined #openstack-infra | 23:26 | |
| *** thorst___ has quit IRC | 23:27 | |
| *** tho______ has joined #openstack-infra | 23:27 | |
| pabelanger | seems to happen as soon as a new node comes online ready | 23:27 |
| *** thor_____ has quit IRC | 23:28 | |
| *** thorst___ has joined #openstack-infra | 23:28 | |
| *** thorst_ has quit IRC | 23:29 | |
| *** IlyaG has joined #openstack-infra | 23:29 | |
| *** thorst_ has joined #openstack-infra | 23:29 | |
| *** thors____ has quit IRC | 23:30 | |
| *** thors____ has joined #openstack-infra | 23:30 | |
| *** Liuqing has quit IRC | 23:30 | |
| *** thorst__ has quit IRC | 23:31 | |
| *** Liuqing has joined #openstack-infra | 23:31 | |
| *** Liuqing has quit IRC | 23:31 | |
| *** thorst__ has joined #openstack-infra | 23:31 | |
| pabelanger | I am unsure how to proceed | 23:31 |
| *** tho______ has quit IRC | 23:32 | |
| *** salv-orl_ has quit IRC | 23:32 | |
| *** Sukhdev has joined #openstack-infra | 23:32 | |
| *** thorst___ has quit IRC | 23:32 | |
| *** thor_____ has joined #openstack-infra | 23:32 | |
| *** thorst___ has joined #openstack-infra | 23:33 | |
| clarkb | I think we try the other things like we discussed, start with foreground and see if that exposes any issues | 23:33 |
| clarkb | and if that doesn't help move on to debugging patches and or reverts | 23:33 |
| *** yamahata has quit IRC | 23:34 | |
| *** tho______ has joined #openstack-infra | 23:34 | |
| pabelanger | clarkb: okay, let me stop again, and launch in the forground | 23:34 |
| *** thors____ has quit IRC | 23:34 | |
| *** watersoul_ has joined #openstack-infra | 23:35 | |
| *** thors____ has joined #openstack-infra | 23:35 | |
| *** thorst__ has quit IRC | 23:36 | |
| *** thorst__ has joined #openstack-infra | 23:36 | |
| pabelanger | started with nodepoold in foreground | 23:36 |
| *** thor_____ has quit IRC | 23:36 | |
| *** thorst_ has quit IRC | 23:37 | |
| *** watersoul has quit IRC | 23:37 | |
| *** thorst_ has joined #openstack-infra | 23:37 | |
| *** thorst___ has quit IRC | 23:37 | |
| *** thorst___ has joined #openstack-infra | 23:38 | |
| *** tho______ has quit IRC | 23:38 | |
| *** amitgandhinz has quit IRC | 23:39 | |
| *** thor_____ has joined #openstack-infra | 23:39 | |
| *** Qiming_ has joined #openstack-infra | 23:39 | |
| *** thors____ has quit IRC | 23:40 | |
| *** thors____ has joined #openstack-infra | 23:40 | |
| *** thorst__ has quit IRC | 23:41 | |
| *** thorst__ has joined #openstack-infra | 23:41 | |
| *** sridhar_ram1 has quit IRC | 23:41 | |
| *** thorst_ has quit IRC | 23:41 | |
| *** cody-somerville has joined #openstack-infra | 23:42 | |
| *** thorst_ has joined #openstack-infra | 23:42 | |
| *** thorst___ has quit IRC | 23:42 | |
| *** thorst___ has joined #openstack-infra | 23:43 | |
| *** thor_____ has quit IRC | 23:43 | |
| *** johnny__ has quit IRC | 23:44 | |
| openstackgerrit | Philip Marc Schwartz proposed openstack-infra/system-config: Add zuul_launcher to openstack_project and run on jenkinsXX hosts. https://review.openstack.org/311165 | 23:44 |
| *** thor_____ has joined #openstack-infra | 23:44 | |
| *** thors____ has quit IRC | 23:45 | |
| *** mriedem has joined #openstack-infra | 23:45 | |
| *** ajo has quit IRC | 23:45 | |
| *** thors____ has joined #openstack-infra | 23:45 | |
| cloudnull | clarkb: I've been beating on the OSIC cluster a bit and I can find anything that looks lke its busted. I've been doing bulk builds and floating IP attachments using heat and all of my instances are coming online timely. | 23:45 |
| clarkb | pabelanger: hoping that with it in the foreground if it is main dying you will see the traceback | 23:45 |
| cloudnull | if you find something please let me know. | 23:45 |
| cloudnull | i'd be happy to dig more into it | 23:45 |
| clarkb | cloudnull: ok, I can get you a list of ips and uuids if thta helps but right now we are not building any new hosts due to a nodepool issue so I don't have new data for you | 23:46 |
| *** thorst__ has quit IRC | 23:46 | |
| *** dmanchad has quit IRC | 23:46 | |
| cloudnull | ok. | 23:46 |
| cloudnull | let me know. | 23:46 |
| *** thorst__ has joined #openstack-infra | 23:46 | |
| pabelanger | clarkb: no, it just locked. Nothing usefully was displayed | 23:46 |
| *** thorst_ has quit IRC | 23:46 | |
| pabelanger | clarkb: I've downgraded paramiko to see if that is the issue | 23:47 |
| pabelanger | clarkb: since things go back once we SSH into a system | 23:47 |
| *** thorst_ has joined #openstack-infra | 23:47 | |
| cloudnull | i should be around this evening and more available tomorrow than i was today | 23:47 |
| pabelanger | clarkb: I think that is the problem, paramiko==2.0.0 | 23:47 |
| pabelanger | clarkb: as nodes are coming back online now | 23:47 |
| *** thorst___ has quit IRC | 23:48 | |
| clarkb | pabelanger: huh | 23:48 |
| pabelanger | clarkb: I think the issue is related when we SSH into the node to run the configure_mirror.sh script | 23:48 |
| *** thorst___ has joined #openstack-infra | 23:48 | |
| pabelanger | but downgrading to paramiko==1.16.0, things are working again | 23:48 |
| clarkb | cloudnull: http://paste.openstack.org/show/496832/ those timestamps are all utc | 23:49 |
| *** ajo has joined #openstack-infra | 23:49 | |
| fungi | we are getting some floating ip quota exceeded exceptions but presumably unrelated | 23:49 |
| fungi | pabelanger: did we end up with paramiko 2.x? | 23:49 |
| *** thor_____ has quit IRC | 23:49 | |
| pabelanger | fungi: ya, I think we'll need to do some cleanups | 23:49 |
| *** thors____ has quit IRC | 23:49 | |
| fungi | that likely came along after the last successful restart | 23:49 |
| pabelanger | fungi: yes | 23:49 |
| fungi | good call | 23:49 |
| pabelanger | so we should likely pin to <2.0 for now | 23:50 |
| pabelanger | until we can debug why it is failing | 23:50 |
| *** thors____ has joined #openstack-infra | 23:50 | |
| *** simondodsley has quit IRC | 23:50 | |
| clarkb | pabelanger: ya if this continues to work for longer than before lets get that in | 23:50 |
| *** thorst__ has quit IRC | 23:50 | |
| *** jamesmca_ has joined #openstack-infra | 23:50 | |
| clarkb | that should also be testable by the devstack plugin | 23:50 |
| *** jamielennox|away is now known as jamielennox | 23:51 | |
| *** thorst__ has joined #openstack-infra | 23:51 | |
| *** dmanchad has joined #openstack-infra | 23:51 | |
| *** thorst_ has quit IRC | 23:52 | |
| *** aratus has quit IRC | 23:52 | |
| *** thor_____ has joined #openstack-infra | 23:52 | |
| *** thorst___ has quit IRC | 23:52 | |
| *** thorst_ has joined #openstack-infra | 23:52 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Update testenvs to use SSD's https://review.openstack.org/315290 | 23:53 |
| *** thorst___ has joined #openstack-infra | 23:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Cap paramiko<2.0.0 https://review.openstack.org/315291 | 23:54 |
| pabelanger | clarkb: fungi: ^ | 23:54 |
| clarkb | pabelanger: if you push a revert of that change up on top of it that should get tested by the functional tests | 23:54 |
| *** jamesmca_ has quit IRC | 23:54 | |
| *** thors____ has quit IRC | 23:54 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Revert "Cap paramiko<2.0.0" https://review.openstack.org/315292 | 23:55 |
| *** thorst__ has quit IRC | 23:55 | |
| *** thorst__ has joined #openstack-infra | 23:56 | |
| pabelanger | we still need to release puppet, which should re-add triple-o | 23:56 |
| *** rvasilets__ has quit IRC | 23:56 | |
| *** thor_____ has quit IRC | 23:56 | |
| *** thors____ has joined #openstack-infra | 23:57 | |
| *** thorst_ has quit IRC | 23:57 | |
| *** thor_____ has joined #openstack-infra | 23:58 | |
| *** thorst___ has quit IRC | 23:58 | |
| *** Qiming_ has quit IRC | 23:58 | |
| *** sflanigan has joined #openstack-infra | 23:58 | |
| *** sflanigan has joined #openstack-infra | 23:58 | |
| *** jsavak has joined #openstack-infra | 23:58 | |
| *** thorst_ has joined #openstack-infra | 23:59 | |
| *** dingyichen has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
