| clarkb | cloudnull: assuming we have enough load you should be able to see those new flavors get used in the mean time | 00:00 |
|---|---|---|
| mtreinish | clarkb, fungi: https://github.com/eclipse/mosquitto/issues/278 | 00:00 |
| fungi | okay, logstash.o.o is online again | 00:00 |
| cloudnull | clarkb: it looks like an s3500 node is in process now :) | 00:00 |
| fungi | mtreinish: that's a beaut | 00:00 |
| cloudnull | so it "should"-tm be working | 00:01 |
| cloudnull | i'll be keeping an eye on things as it gets spinning | 00:01 |
| *** aeng__ has joined #openstack-infra | 00:01 | |
| clarkb | cloudnull: sounds good | 00:01 |
| cloudnull | thanks again eveyone for all the assistance. | 00:02 |
| *** aeng_ has quit IRC | 00:02 | |
| fungi | clarkb: tailing /var/log/logprocessor/log-client-debug.log and so far just one line: 2016-09-28 00:00:29,813 Log pusher starting. | 00:02 |
| fungi | so it's not like it's going crazy at the moment at least | 00:02 |
| *** thcipriani is now known as thcipriani|afk | 00:02 | |
| *** roxanaghe has quit IRC | 00:02 | |
| *** thorst has joined #openstack-infra | 00:03 | |
| clarkb | also some deduping of images did happen despite the races | 00:03 |
| clarkb | fungi: you should see things generated theer when jobs start finishing | 00:03 |
| *** spzala_ has quit IRC | 00:04 | |
| *** claudiub|2 has quit IRC | 00:04 | |
| *** mriedem has joined #openstack-infra | 00:04 | |
| fungi | yeah, i expect just no jobs have completed yet | 00:04 |
| clarkb | doesn't look like the process is stillrunning | 00:04 |
| *** sflanigan_ has quit IRC | 00:04 | |
| clarkb | oh! | 00:04 |
| fungi | oh, huh... | 00:04 |
| fungi | stale pidfile? | 00:04 |
| clarkb | we might still be affected by the thing where dns isn't working when the service starts so it dies | 00:04 |
| clarkb | but maybe also a stale pid file | 00:05 |
| fungi | aha | 00:05 |
| fungi | i'll start it manually and make sure it keeps going | 00:05 |
| clarkb | upstart + sysv init scripts makes this very hard | 00:05 |
| clarkb | fungi: ok | 00:05 |
| *** pahuang__ has quit IRC | 00:05 | |
| jlvillal | Any chance we can get a review on https://review.openstack.org/#/c/352909/ It already has two +2 votes. This is a devstack-gate patch. Trying to get Ironic multi-node testing going. | 00:05 |
| clarkb | I should just make systemd unit files for them then we can upgrade them to xenial real soon once precise is gone | 00:05 |
| *** thorst has quit IRC | 00:05 | |
| anteaya | fungi: thanks for reapplying your +1 to 376828 after I rebased, I appreciate it | 00:05 |
| fungi | clarkb: yeah, i found no stale pidfile anyway | 00:05 |
| *** xarses has quit IRC | 00:06 | |
| *** aeng__ has quit IRC | 00:06 | |
| fungi | the log is definitely active now | 00:06 |
| fungi | _very_ active | 00:06 |
| *** zhurong has quit IRC | 00:06 | |
| fungi | 200k lines of active just in the first minute since starting it | 00:07 |
| fungi | maybe this is a little _too_ debug? | 00:07 |
| clarkb | ya we might want to consider pulling back on how much we log there | 00:07 |
| clarkb | though I don't know why it skyrocketed in july | 00:07 |
| clarkb | oh maybe that was when we put zuul launchers in place? | 00:08 |
| fungi | oh yeah, almost up to 50mib already | 00:08 |
| clarkb | and that is basically directly correlated to traffic from the job launchers whether zuul or jenkins | 00:08 |
| clarkb | jlvillal: done | 00:09 |
| jlvillal | clarkb: Thank you :) | 00:09 |
| clarkb | jlvillal: out of curiousity was that causing failure because c-vol/c-bak fail to start if no c-api is running? | 00:10 |
| clarkb | (I always thought our services were supposed to be resilient to that...) | 00:10 |
| jlvillal | clarkb: I believe so. On the Ironic testing we don't have any cinder at all. | 00:10 |
| jlvillal | clarkb: So when the subnode would try to start it, it would fail. | 00:11 |
| *** davidlenwell has quit IRC | 00:11 | |
| *** edmondsw has quit IRC | 00:11 | |
| fungi | clarkb: so with zuul mostly idle at the moment, the log client is recording an average of 300kib of debug logs a second. if my math is correct, it will fill up the rootfs again at this rate in 34 hours time | 00:12 |
| clarkb | fungi: fun | 00:13 |
| *** amitgandhinz has joined #openstack-infra | 00:13 | |
| *** ddieterly has quit IRC | 00:13 | |
| *** ijw has quit IRC | 00:14 | |
| *** tpsilva has quit IRC | 00:14 | |
| clarkb | fungi: quick interim fix would be to remove the -d/--debuglog from the command invocation | 00:14 |
| *** kaisers2 has joined #openstack-infra | 00:15 | |
| clarkb | fungi: I can push taht patch in a sec | 00:15 |
| fungi | ahh, thanks. starting to run out of steam over here | 00:16 |
| *** kaisers3 has joined #openstack-infra | 00:16 | |
| *** kaisers has quit IRC | 00:17 | |
| *** kaisers1 has quit IRC | 00:17 | |
| *** aeng__ has joined #openstack-infra | 00:17 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-log_processor: Reduce log client logging by default https://review.openstack.org/378104 | 00:18 |
| clarkb | fungi: ^ that should do it | 00:18 |
| clarkb | though we should write a better long term logging setup for it | 00:18 |
| *** ijw has joined #openstack-infra | 00:19 | |
| *** sflanigan_ has joined #openstack-infra | 00:19 | |
| clarkb | cloudnull: ok all iamges are uploaded to osic's logical providers with some deduping too | 00:19 |
| *** markvoelker has joined #openstack-infra | 00:20 | |
| cloudnull | so the image upload skip bits are working as expected? | 00:20 |
| clarkb | for the most part | 00:21 |
| cloudnull | thats great ! | 00:21 |
| clarkb | its not 100% since they can race each other but where the racing happens the way we want it looks like it works | 00:21 |
| cloudnull | I see the new images look like "nodepool-ubuntu-xenial-1160" | 00:21 |
| *** pahuang__ has joined #openstack-infra | 00:21 | |
| cloudnull | and the old ones were "template-debian-jessie-1474978264" | 00:21 |
| cloudnull | is that right ? | 00:22 |
| clarkb | cloudnull: yup | 00:22 |
| cloudnull | then it's lookin good on my end too. | 00:22 |
| cloudnull | :) | 00:22 |
| clarkb | cloudnull: also left a comment on your new change. its not quite right | 00:22 |
| cloudnull | ok | 00:22 |
| * cloudnull goes to look | 00:22 | |
| clarkb | cloudnull: basically we have to leave the old provider in place with a max server value of inactive | 00:22 |
| clarkb | so that nodepool can gracefully clean up the resources it associates with that provider | 00:22 |
| clarkb | jeblair: when you return tomorrow I also think we might have a race on the delete side ... that one could get a bit nasty | 00:23 |
| clarkb | jeblair: basically with this new dedup stuff we want to treat deletes on the whole of the deduped images | 00:23 |
| clarkb | otherwise we could remove the image from one provider and not realize another provider is using it and leads to boot fails | 00:23 |
| clarkb | this can happen if the upload time between providers is long (it shouldn't be so likely won't be a problem in practice) | 00:24 |
| * clarkb likes problems that are not problems in practice | 00:24 | |
| clarkb | like the added cancer risk of going to mars according to elon musk | 00:24 |
| clarkb | "it will be higher but not that much higher" | 00:24 |
| cloudnull | so should I set it to "inactive" or 0? | 00:25 |
| *** benk01 has joined #openstack-infra | 00:25 | |
| clarkb | cloudnull: to -1 | 00:25 |
| benk01 | Hello. | 00:25 |
| *** markvoelker has quit IRC | 00:25 | |
| clarkb | max-servers: -1 | 00:25 |
| *** davidlenwell has joined #openstack-infra | 00:25 | |
| clarkb | benk01: hello | 00:25 |
| cloudnull | and should I leave the image entries in this case ? | 00:25 |
| cloudnull | the -name: ...\n... - name: osic-cloud1 ? | 00:26 |
| clarkb | cloudnull: under labels you mean? i think you can continue to remove them but its not necessary unless we completely remove the provider later | 00:26 |
| cloudnull | ok | 00:27 |
| benk01 | Hey guys, my review.openstack.org account is in a state of flux. I ran through the "Agreementes" section; though left my address details bare. It seems these details are needed in order to submit commits to gerrit and when I try and enter such details I get "server error: cannot store contact information" | 00:28 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Update server distribution to HW specific entries https://review.openstack.org/378100 | 00:28 |
| benk01 | like; I have done my ICLA, though I was prompted later to fill in contact info. | 00:28 |
| clarkb | yes in order to properly sign the cla you need to include contact info | 00:28 |
| *** xarses has joined #openstack-infra | 00:29 | |
| cloudnull | so i put it all back and just changed the max-server settings. If we need to remove the provider all together I'll cross that bridge when I get there. | 00:29 |
| clarkb | cloudnull: ok | 00:30 |
| *** adrian_otto has quit IRC | 00:30 | |
| cloudnull | unless you think its best to remove those labels now . | 00:30 |
| benk01 | clarkb: how do I complete the ICLA/contact information storage? review.openstack keeps throwing errors when I try to store it. | 00:30 |
| fungi | benk01: your foundation profile primary e-mail address at www.openstack.org needs to match your preferred e-mail address in gerrit's contact settings | 00:31 |
| *** tqtran has quit IRC | 00:31 | |
| fungi | the contact storing form in gerrit is an api callback to the foundation member system to update the linkage against your foundation profile | 00:31 |
| clarkb | cloudnull: nah its fine to deal with it when we get there | 00:32 |
| cloudnull | ok | 00:32 |
| benk01 | oh ok. I don't think I have a foundation profile yet. I will create one now. | 00:32 |
| *** markvoelker has joined #openstack-infra | 00:34 | |
| kzaitsev_mb | what should I do if I have a project in glaobl-requirements and all the jenkins jobs fail with Could not satisfy constraints for 'murano-pkg-check': installation from path or url cannot be constrained to a version | 00:35 |
| kzaitsev_mb | there must be a workaround that is used by clients and oslo libs, right? | 00:35 |
| *** gordc has joined #openstack-infra | 00:35 | |
| cloudnull | clarkb: it looks like image 9b34985b-a92e-42cb-8dbe-29d39f54c06d "nodepool-ubuntu-xenial-1160" is queued and has been for a while. maybe the upload didn't complete there? | 00:36 |
| clarkb | hrm nodepool thinks they all finished, I winder if that got lost in my restart of the builder | 00:39 |
| fungi | benk01: http://docs.openstack.org/infra/manual/developers.html#account-setup is what we normally expect new users to find (it's linked from lots of places) | 00:39 |
| clarkb | cloudnull: does it look like it is hurting anything? if not I think I will debug in the morning | 00:39 |
| cloudnull | nope its now | 00:39 |
| clarkb | so I canstart on evening things now | 00:39 |
| cloudnull | *not | 00:39 |
| cloudnull | so cheers, and have a good night ! | 00:40 |
| clarkb | you too | 00:40 |
| *** caowei has joined #openstack-infra | 00:40 | |
| *** Douhet has joined #openstack-infra | 00:42 | |
| *** doffm has quit IRC | 00:42 | |
| *** adrian_otto has joined #openstack-infra | 00:42 | |
| *** doffm has joined #openstack-infra | 00:42 | |
| *** gyee has quit IRC | 00:43 | |
| *** nijaba has quit IRC | 00:44 | |
| *** ijw has quit IRC | 00:45 | |
| *** Apoorva_ has joined #openstack-infra | 00:45 | |
| *** nijaba has joined #openstack-infra | 00:45 | |
| *** nijaba has joined #openstack-infra | 00:45 | |
| *** bin_ has quit IRC | 00:46 | |
| *** ddieterly has joined #openstack-infra | 00:46 | |
| *** bin_ has joined #openstack-infra | 00:46 | |
| *** adrian_otto has quit IRC | 00:46 | |
| *** adrian_otto has joined #openstack-infra | 00:47 | |
| *** psilvad has quit IRC | 00:47 | |
| benk01 | Thanks; I think my account is now set up. | 00:47 |
| *** amitgandhinz has quit IRC | 00:47 | |
| *** salv-orlando has joined #openstack-infra | 00:48 | |
| *** Apoorva has quit IRC | 00:48 | |
| *** Apoorva_ has quit IRC | 00:50 | |
| *** tuannguyen has joined #openstack-infra | 00:50 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 00:52 | |
| *** salv-orlando has quit IRC | 00:52 | |
| *** tuannguyen has quit IRC | 00:54 | |
| *** M-docaedo_vector has joined #openstack-infra | 00:58 | |
| *** ddieterly has quit IRC | 01:01 | |
| dims | anyone with git admin privs around? i typo'ed a branch creation | 01:02 |
| *** kvcobb has quit IRC | 01:04 | |
| fungi | i can clean it up. what do you need? | 01:04 |
| *** benk01 has quit IRC | 01:04 | |
| *** Goneri has quit IRC | 01:04 | |
| fungi | dims: ^ | 01:05 |
| dims | ah, one sec | 01:05 |
| *** ijw has joined #openstack-infra | 01:05 | |
| dims | fungi : need to nuke http://git.openstack.org/cgit/openstack/tacker/log/?h=stable/newton, i typoed the branch creation of a mitaka SHA | 01:06 |
| armax | hi folks sorry for the (silly) question | 01:06 |
| armax | but how does one retire a project? | 01:06 |
| *** adrian_otto1 has joined #openstack-infra | 01:06 | |
| dims | fungi : thanks a ton! | 01:06 |
| dims | armax : http://ronaldbradford.com/blog/retiring-an-openstack-project-2016-03-28/ | 01:06 |
| armax | http://docs.openstack.org/infra/manual/drivers.html#retiring-a-project? | 01:06 |
| fungi | armax: http://docs.openstack.org/infra/manual/drivers.html#retiring-a-project | 01:06 |
| dims | armax : y that blog has a link to http://docs.openstack.org/infra/manual/drivers.html#retiring-a-project | 01:06 |
| armax | dims: ack | 01:07 |
| armax | I am thinking of retiring neutron | 01:07 |
| armax | thanks dims :) | 01:07 |
| fungi | armax: you're going to get some people excited if you say that ;) | 01:07 |
| dims | armax : haha back to nova-network? :) | 01:07 |
| armax | no, no networking at all | 01:08 |
| armax | networking is overrated anyway | 01:08 |
| * dims hands out pitchforks | 01:08 | |
| *** adrian_otto has quit IRC | 01:08 | |
| dims | armax : ooh, i neeed your +1 in https://review.openstack.org/#/c/374820/ | 01:09 |
| armax | dims: sure thing | 01:09 |
| dims | thank you sir! | 01:09 |
| armax | looking | 01:09 |
| harlowja | no more networking ++ | 01:09 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add c-vol,c-bak on subnode when c-api enabled https://review.openstack.org/352909 | 01:10 |
| fungi | dims: i can't delete it--there are open changes for that branch that need to get merged or abandoned first | 01:10 |
| *** rvba` has quit IRC | 01:10 | |
| *** ijw_ has joined #openstack-infra | 01:10 | |
| dims | one sec let me see if i can find those folks | 01:10 |
| fungi | armax: maybe we can use cinder to tunnel ip over fiber channel? (like a reverse of iscsi?) | 01:10 |
| *** senk_ has joined #openstack-infra | 01:11 | |
| armax | fungi: I am sure that’s a lot less scary than what neutron has become | 01:11 |
| harlowja | pigeons | 01:11 |
| *** ijw has quit IRC | 01:11 | |
| dims | fungi : clean now https://review.openstack.org/#/q/project:openstack/tacker+branch:stable/newton | 01:11 |
| dims | it was just my .gitreview change | 01:11 |
| fungi | dims: i've deleted stable/newton for openstack/tacker, formerly at 04a973be4e5106c0bf001de9054a46a4d7a88463 | 01:12 |
| *** rvba has joined #openstack-infra | 01:12 | |
| *** rvba has quit IRC | 01:12 | |
| *** rvba has joined #openstack-infra | 01:12 | |
| dims | harlowja : while we are at it nova too? (see https://github.com/Mirantis/virtlet) | 01:12 |
| dims | fungi : thanks a ton for the quick response! | 01:12 |
| harlowja | dims hmmm, i was wondering when something like that would appear | 01:12 |
| dims | armax harlowja - am kidding of course | 01:12 |
| harlowja | ha | 01:12 |
| *** kaisers__ has quit IRC | 01:12 | |
| harlowja | ha | 01:12 |
| harlowja | :-/ | 01:12 |
| dims | harlowja : it's a experiment :) | 01:12 |
| harlowja | :) | 01:13 |
| harlowja | ya, its hard for me to decide if k8s is just openstack at a much earlier phase in its life (except with containers!) | 01:13 |
| harlowja | and with a slightly bigger backer | 01:14 |
| fungi | containers and pixy dust | 01:14 |
| *** gildub has quit IRC | 01:14 | |
| harlowja | yup | 01:14 |
| *** gongysh has joined #openstack-infra | 01:14 | |
| harlowja | but dims ya i expect virtlet or something like it soonish | 01:14 |
| harlowja | its a natural question of 'why just containers' | 01:14 |
| *** senk_ has quit IRC | 01:15 | |
| dims | harlowja : fungi : right | 01:15 |
| *** ijw_ has quit IRC | 01:15 | |
| harlowja | life is like a big circle | 01:16 |
| harlowja | lol | 01:16 |
| * fungi thinks time sharing systems are making a comeback | 01:16 | |
| dims | :) | 01:16 |
| harlowja | not yet, but soon, lol | 01:17 |
| harlowja | out for the day, peace | 01:17 |
| dims | good evening harlowja | 01:18 |
| *** zhurong has joined #openstack-infra | 01:18 | |
| *** rossella_ has quit IRC | 01:19 | |
| *** gordc has quit IRC | 01:19 | |
| *** rossella_ has joined #openstack-infra | 01:19 | |
| *** gildub has joined #openstack-infra | 01:20 | |
| dims | armax : thanks for the +1, agree on the add/revert of the workaround | 01:21 |
| armax | ack | 01:22 |
| *** lezbar has quit IRC | 01:22 | |
| *** lezbar has joined #openstack-infra | 01:23 | |
| *** pkoniszewski has quit IRC | 01:25 | |
| *** pkoniszewski has joined #openstack-infra | 01:31 | |
| *** thorst has joined #openstack-infra | 01:32 | |
| *** thorst has quit IRC | 01:32 | |
| *** yanyanhu_ has joined #openstack-infra | 01:38 | |
| *** lezbar has quit IRC | 01:40 | |
| *** lezbar has joined #openstack-infra | 01:40 | |
| *** amitgandhinz has joined #openstack-infra | 01:44 | |
| dims | fungi : please nuke http://git.openstack.org/cgit/openstack/tacker-horizon/log/?h=stable/newton as well. that's in the same typo-ed batch | 01:46 |
| *** tonytan4ever has joined #openstack-infra | 01:46 | |
| *** rushil has joined #openstack-infra | 01:47 | |
| *** ddieterly has joined #openstack-infra | 01:49 | |
| *** tonytan4ever has quit IRC | 01:50 | |
| *** kzaitsev_mb has quit IRC | 01:51 | |
| *** ddieterly has quit IRC | 01:52 | |
| *** hongbin has joined #openstack-infra | 01:54 | |
| *** gema has quit IRC | 01:56 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Fix typo in openstack_functions.py https://review.openstack.org/378143 | 01:57 |
| *** Jeffrey4l has joined #openstack-infra | 01:58 | |
| *** kmartin has quit IRC | 02:02 | |
| *** gema has joined #openstack-infra | 02:03 | |
| *** liusheng has quit IRC | 02:03 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project zaqar https://review.openstack.org/378143 | 02:04 |
| *** liusheng has joined #openstack-infra | 02:04 | |
| *** tphummel has quit IRC | 02:04 | |
| *** baoli has quit IRC | 02:05 | |
| *** baoli has joined #openstack-infra | 02:11 | |
| *** tuannguyen has joined #openstack-infra | 02:12 | |
| *** amitgandhinz has quit IRC | 02:18 | |
| *** ijw has joined #openstack-infra | 02:19 | |
| *** ddieterly has joined #openstack-infra | 02:21 | |
| *** ijw has quit IRC | 02:23 | |
| *** ijw has joined #openstack-infra | 02:24 | |
| *** rlandy has quit IRC | 02:24 | |
| *** ddieterly has quit IRC | 02:25 | |
| *** adrian_otto1 has quit IRC | 02:25 | |
| *** sflanigan_ has quit IRC | 02:26 | |
| *** ijw has quit IRC | 02:29 | |
| *** mriedem has quit IRC | 02:30 | |
| *** ijw has joined #openstack-infra | 02:31 | |
| *** salv-orlando has joined #openstack-infra | 02:36 | |
| *** sflanigan_ has joined #openstack-infra | 02:39 | |
| *** salv-orlando has quit IRC | 02:40 | |
| *** shu-mutou has quit IRC | 02:44 | |
| *** ijw has quit IRC | 02:44 | |
| *** shu-mutou has joined #openstack-infra | 02:44 | |
| *** baoli has quit IRC | 02:52 | |
| *** lezbar has quit IRC | 02:56 | |
| *** lezbar has joined #openstack-infra | 02:57 | |
| *** Sukhdev has joined #openstack-infra | 02:59 | |
| *** tuanla has joined #openstack-infra | 03:00 | |
| *** Sukhdev has quit IRC | 03:01 | |
| *** david-lyle has quit IRC | 03:04 | |
| *** spzala has joined #openstack-infra | 03:05 | |
| *** spzala has quit IRC | 03:05 | |
| *** hongbin has quit IRC | 03:10 | |
| *** adrian_otto has joined #openstack-infra | 03:12 | |
| *** rfolco_ has quit IRC | 03:14 | |
| *** amitgandhinz has joined #openstack-infra | 03:14 | |
| *** sdake_ has quit IRC | 03:17 | |
| *** armax has quit IRC | 03:18 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack/os-client-config: Using assertIsNone() instead of assertEqual(None, ...) https://review.openstack.org/378205 | 03:25 |
| *** adrian_otto has quit IRC | 03:26 | |
| *** jamesden_ has quit IRC | 03:26 | |
| *** adrian_otto has joined #openstack-infra | 03:26 | |
| *** vikrant has joined #openstack-infra | 03:26 | |
| *** tuannguyen has quit IRC | 03:29 | |
| *** tqtran has joined #openstack-infra | 03:30 | |
| *** rajinir has quit IRC | 03:35 | |
| *** tqtran has quit IRC | 03:36 | |
| *** roxanaghe has joined #openstack-infra | 03:38 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Updated grafana to support new OSIC HW specs https://review.openstack.org/378215 | 03:38 |
| *** sdake has joined #openstack-infra | 03:39 | |
| cloudnull | So in osic-cloud1 it looks like we're missing the xenial image from tonights upload. I know folks are likely AFK now but when possible if we can get that image re-uploaded I think it'll make our error node count decline. | 03:44 |
| irtermite | :( | 03:44 |
| cloudnull | I don't know what's specifically causing http://grafana.openstack.org/dashboard/db/nodepool-osic to show error node launch attempts but i guessing it's because of that image missing. | 03:45 |
| *** amitgandhinz has quit IRC | 03:49 | |
| *** sdake_ has joined #openstack-infra | 03:50 | |
| *** Sukhdev has joined #openstack-infra | 03:50 | |
| *** sdake has quit IRC | 03:51 | |
| *** roxanaghe has quit IRC | 03:52 | |
| *** adrian_otto has quit IRC | 03:53 | |
| *** tongli_ has quit IRC | 03:57 | |
| *** gongysh has quit IRC | 03:58 | |
| *** roxanaghe has joined #openstack-infra | 03:59 | |
| *** njohnston__ is now known as njohnston | 04:01 | |
| *** ramishra has quit IRC | 04:03 | |
| *** sflanigan_ has quit IRC | 04:04 | |
| *** rkukura has quit IRC | 04:05 | |
| *** ramishra has joined #openstack-infra | 04:07 | |
| *** yamamoto_ has quit IRC | 04:17 | |
| *** sflanigan_ has joined #openstack-infra | 04:20 | |
| *** pabelanger has quit IRC | 04:22 | |
| *** sai has quit IRC | 04:22 | |
| *** roxanaghe has quit IRC | 04:23 | |
| *** sai has joined #openstack-infra | 04:23 | |
| *** pabelanger has joined #openstack-infra | 04:23 | |
| *** amotoki has joined #openstack-infra | 04:28 | |
| *** claudiub|2 has joined #openstack-infra | 04:31 | |
| *** Rocky_g has quit IRC | 04:36 | |
| *** psachin has joined #openstack-infra | 04:41 | |
| *** amitgandhinz has joined #openstack-infra | 04:45 | |
| *** gongysh has joined #openstack-infra | 04:49 | |
| *** yamamoto has joined #openstack-infra | 04:59 | |
| *** rkukura has joined #openstack-infra | 05:00 | |
| *** pgadiya has joined #openstack-infra | 05:07 | |
| *** Sukhdev has quit IRC | 05:08 | |
| *** links has joined #openstack-infra | 05:09 | |
| *** links has quit IRC | 05:15 | |
| *** links has joined #openstack-infra | 05:17 | |
| *** salv-orlando has joined #openstack-infra | 05:18 | |
| *** rossella_ has quit IRC | 05:19 | |
| *** rossella_ has joined #openstack-infra | 05:19 | |
| *** amitgandhinz has quit IRC | 05:20 | |
| *** sshnaidm|afk is now known as sshnaidm | 05:21 | |
| *** salv-orlando has quit IRC | 05:22 | |
| *** prometheanfire has quit IRC | 05:23 | |
| *** gildub has quit IRC | 05:26 | |
| *** gildub has joined #openstack-infra | 05:28 | |
| *** nadya has joined #openstack-infra | 05:29 | |
| *** woodster_ has quit IRC | 05:30 | |
| *** prometheanfire has joined #openstack-infra | 05:30 | |
| *** sdake_ has quit IRC | 05:30 | |
| *** nadya has quit IRC | 05:31 | |
| *** flepied has quit IRC | 05:32 | |
| *** ssancheztrujillo has joined #openstack-infra | 05:32 | |
| *** apetrich has quit IRC | 05:32 | |
| *** apetrich has joined #openstack-infra | 05:37 | |
| *** camunoz_ has joined #openstack-infra | 05:38 | |
| *** kaisers has joined #openstack-infra | 05:39 | |
| *** jaosorior has joined #openstack-infra | 05:46 | |
| *** nherciu has joined #openstack-infra | 05:53 | |
| *** j_king has quit IRC | 05:53 | |
| *** rushil has quit IRC | 05:59 | |
| *** yuval has joined #openstack-infra | 06:00 | |
| ramishra | Hi guys, we're seeing numerous dsvm gate job failures with error connecting to the fedora mirrors (ex. Failed to connect to mirrors.mit.edu port 80: Connection timed out). Are there any infra network issues atm? | 06:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix upstream tracking repos https://review.openstack.org/378016 | 06:08 |
| *** rcernin has joined #openstack-infra | 06:14 | |
| *** nadya has joined #openstack-infra | 06:14 | |
| *** amitgandhinz has joined #openstack-infra | 06:16 | |
| *** nadya has quit IRC | 06:18 | |
| *** andreas_s has joined #openstack-infra | 06:24 | |
| *** danielitit has joined #openstack-infra | 06:27 | |
| *** aeng__ has quit IRC | 06:30 | |
| *** aeng__ has joined #openstack-infra | 06:30 | |
| *** crinkle_ is now known as crinkle | 06:33 | |
| *** sflanigan_ has quit IRC | 06:34 | |
| openstackgerrit | Robin Naundorf proposed openstack-infra/system-config: Fix small typos in docs https://review.openstack.org/377691 | 06:35 |
| *** mrunge has quit IRC | 06:36 | |
| *** andreas_s has quit IRC | 06:36 | |
| *** Ravikiran_K has joined #openstack-infra | 06:36 | |
| *** pcaruana has joined #openstack-infra | 06:37 | |
| *** flepied has joined #openstack-infra | 06:37 | |
| *** ihrachys has joined #openstack-infra | 06:37 | |
| *** andreas_s has joined #openstack-infra | 06:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Enable layout-checks.py again https://review.openstack.org/378275 | 06:39 |
| *** ihrachys has quit IRC | 06:40 | |
| *** ihrachys has joined #openstack-infra | 06:41 | |
| *** mrunge has joined #openstack-infra | 06:42 | |
| openstackgerrit | Robin Naundorf proposed openstack-infra/system-config: Fix deadlink in wiki doc https://review.openstack.org/378280 | 06:43 |
| *** ihrachys has quit IRC | 06:44 | |
| *** sflanigan_ has joined #openstack-infra | 06:45 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: make upgrade jobs voting https://review.openstack.org/378284 | 06:48 |
| AJaeger | project-config cores, could you review two changes to fix track-upstream repos, please? https://review.openstack.org/378003 and https://review.openstack.org/378016 | 06:49 |
| *** amitgandhinz has quit IRC | 06:50 | |
| *** salv-orlando has joined #openstack-infra | 06:50 | |
| *** drifterza has joined #openstack-infra | 06:54 | |
| *** isaacb has joined #openstack-infra | 06:55 | |
| *** senk_ has joined #openstack-infra | 06:56 | |
| *** pahuang__ has quit IRC | 06:56 | |
| senk_ | i guess something is not working on this one https://review.openstack.org/#/c/329410/ | 06:57 |
| AJaeger | senk_: see "Related changes" - it's dependent on two other changes. Those need to merge first. | 06:58 |
| senk_ | ah okay thank you for pointing me to that AJaeger :) | 06:58 |
| *** muawiakhan has joined #openstack-infra | 06:59 | |
| *** tuanla has quit IRC | 07:01 | |
| *** [HeOS] has joined #openstack-infra | 07:02 | |
| *** gildub has quit IRC | 07:04 | |
| *** amoralej|off is now known as amoralej | 07:04 | |
| yolanda | good morning | 07:05 |
| *** ralonsoh has joined #openstack-infra | 07:06 | |
| therve | AJaeger, we're getting awful external connectivity in the gate in the past 2 days, I don't know if there is something going on | 07:08 |
| *** pahuang__ has joined #openstack-infra | 07:08 | |
| AJaeger | morning, yolanda ! | 07:09 |
| *** gildub has joined #openstack-infra | 07:09 | |
| therve | Mostly OSIC cloud apparently, though it seems to be running most of our jobs | 07:09 |
| AJaeger | therve: that's sad - could you summarize on openstack-infra, please? This needs cloudnull and the US based infra-roots. | 07:09 |
| *** lezbar has quit IRC | 07:10 | |
| AJaeger | therve: there's some work on changing OSIC setup going on, but reference to a few jobs/log files that show the behaviour would be great | 07:10 |
| *** matrohon has joined #openstack-infra | 07:10 | |
| therve | AJaeger, WDYM my openstack-infra? | 07:10 |
| *** lezbar has joined #openstack-infra | 07:10 | |
| AJaeger | therve: http://lists.openstack.org/pipermail/openstack-infra/ | 07:11 |
| therve | OK sure | 07:11 |
| AJaeger | therve: I should have mentioned *mailing list* | 07:11 |
| therve | No worries :) | 07:11 |
| AJaeger | yolanda: when you review, please prioritize https://review.openstack.org/377738 https://review.openstack.org/377947 https://review.openstack.org/378003 and https://review.openstack.org/378016 | 07:12 |
| *** links has quit IRC | 07:16 | |
| yolanda | hi sure | 07:17 |
| AJaeger | thanks, yolanda | 07:24 |
| *** camunoz_ has quit IRC | 07:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Move periodic-python-jobs-with-neutron-lib-master to xenial https://review.openstack.org/377738 | 07:31 |
| *** jlanoux has joined #openstack-infra | 07:33 | |
| *** winggundamth has joined #openstack-infra | 07:33 | |
| *** tuannguyen has joined #openstack-infra | 07:33 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: test upgrade from newton branch https://review.openstack.org/378305 | 07:34 |
| *** nmagnezi has joined #openstack-infra | 07:35 | |
| *** jpich has joined #openstack-infra | 07:35 | |
| *** tuannguyen has quit IRC | 07:39 | |
| *** jpena|off is now known as jpena | 07:40 | |
| AJaeger | dims, http://logs.openstack.org/periodic/periodic-nova-py27-with-oslo-master/32d4c97/console.html is failing every time - looks like the tox.ini generated does not work. Could you check and fix it? Or should we remove it since nobody cares? | 07:40 |
| *** gildub has quit IRC | 07:42 | |
| *** salv-orl_ has joined #openstack-infra | 07:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Move py34-with-oslo-master jobs to xenial https://review.openstack.org/377947 | 07:44 |
| *** links has joined #openstack-infra | 07:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Open a repository for NFV filters. https://review.openstack.org/377794 | 07:45 |
| openstackgerrit | Merged openstack-infra/project-config: gnocchi: make upgrade jobs voting https://review.openstack.org/378284 | 07:45 |
| *** salv-orlando has quit IRC | 07:46 | |
| *** amitgandhinz has joined #openstack-infra | 07:46 | |
| *** gildub has joined #openstack-infra | 07:48 | |
| *** strigazi_AFK is now known as strigazi | 07:49 | |
| *** pilgrimstack has joined #openstack-infra | 07:50 | |
| *** sshnaidm has quit IRC | 07:51 | |
| *** ihrachys has joined #openstack-infra | 07:51 | |
| *** nadya has joined #openstack-infra | 07:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable py35 voting on project zaqar https://review.openstack.org/378143 | 07:53 |
| openstackgerrit | Merged openstack-infra/project-config: Add checks to openstack/cookbook-openstack-application-catalog https://review.openstack.org/377671 | 07:53 |
| openstackgerrit | Merged openstack-infra/project-config: Create puppet-cloudkitty repository https://review.openstack.org/364535 | 07:55 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: WIP: Allow permissions to be set for teams in worklists and boards https://review.openstack.org/370312 | 07:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Allow permissions to be set for teams on stories https://review.openstack.org/368917 | 07:58 |
| *** zzzeek has quit IRC | 08:00 | |
| *** yanyanhu_ has quit IRC | 08:01 | |
| *** hashar has joined #openstack-infra | 08:01 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** sc68cal_ has joined #openstack-infra | 08:02 | |
| *** ijw has joined #openstack-infra | 08:03 | |
| *** Na3iL has joined #openstack-infra | 08:03 | |
| *** yaume has joined #openstack-infra | 08:03 | |
| *** sc68cal has quit IRC | 08:03 | |
| *** yanyanhu_ has joined #openstack-infra | 08:05 | |
| *** M-docaedo_vector has quit IRC | 08:05 | |
| *** [HeOS] is now known as HeOS | 08:09 | |
| *** watersoul has quit IRC | 08:11 | |
| *** ijw has quit IRC | 08:11 | |
| *** eharney has quit IRC | 08:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch ironic-inspector to tempest discovery job https://review.openstack.org/377623 | 08:19 |
| openstackgerrit | Merged openstack-infra/project-config: Convert tripleo-container jobs into multinode https://review.openstack.org/377000 | 08:20 |
| *** esikachev has joined #openstack-infra | 08:21 | |
| *** links has quit IRC | 08:21 | |
| *** gildub has quit IRC | 08:21 | |
| *** amitgandhinz has quit IRC | 08:21 | |
| *** IzikP has quit IRC | 08:32 | |
| *** tqtran has joined #openstack-infra | 08:34 | |
| *** hferenc has joined #openstack-infra | 08:35 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add openstackci oscc cloud to Puppetmaster https://review.openstack.org/378384 | 08:36 |
| *** akijak has left #openstack-infra | 08:37 | |
| *** links has joined #openstack-infra | 08:37 | |
| *** tqtran has quit IRC | 08:38 | |
| *** sdake has joined #openstack-infra | 08:41 | |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** sdake has quit IRC | 08:42 | |
| *** sshnaidm has joined #openstack-infra | 08:42 | |
| *** jed56 has joined #openstack-infra | 08:45 | |
| *** chem has joined #openstack-infra | 08:45 | |
| *** mhickey has joined #openstack-infra | 08:46 | |
| *** ssancheztrujillo has quit IRC | 08:47 | |
| *** priteau has joined #openstack-infra | 08:48 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #openstack-infra | 08:49 | |
| *** claudiub|2 has quit IRC | 08:52 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/irc-meetings: Update Horizon Meeting Chair to new PTL https://review.openstack.org/378417 | 08:54 |
| *** _oanson is now known as oanson | 08:55 | |
| *** teran has quit IRC | 08:56 | |
| *** kaisers3 has quit IRC | 08:57 | |
| *** Julien-z_ has joined #openstack-infra | 08:57 | |
| *** Julien-z_ has quit IRC | 08:57 | |
| *** Julien-z_ has joined #openstack-infra | 08:58 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Make Horizon and D_O_A Django 1.10 tests voting https://review.openstack.org/378426 | 08:58 |
| *** dtardivel has joined #openstack-infra | 08:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:58 | |
| *** kaisers1 has joined #openstack-infra | 08:59 | |
| *** jamielennox is now known as jamielennox|away | 08:59 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack-infra/project-config: Add new project: devstack-plugin-libvirt-qemu https://review.openstack.org/377920 | 09:02 |
| *** watersoul has joined #openstack-infra | 09:06 | |
| openstackgerrit | Merged openstack-infra/system-config: Add openstackci oscc cloud to Puppetmaster https://review.openstack.org/378384 | 09:07 |
| *** claudiub|2 has joined #openstack-infra | 09:11 | |
| *** sambetts_ is now known as sambetts | 09:12 | |
| *** kaisers has quit IRC | 09:12 | |
| *** e0ne has joined #openstack-infra | 09:14 | |
| *** OferBarber has joined #openstack-infra | 09:15 | |
| *** pgadiya has quit IRC | 09:15 | |
| *** priteau has quit IRC | 09:15 | |
| Guest75624 | Where can I find results for Post jobs - e.g. the coverage jobs? | 09:16 |
| *** priteau has joined #openstack-infra | 09:17 | |
| *** dmellado_ is now known as dmellado | 09:18 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate openstackci launcher layouts https://review.openstack.org/378458 | 09:18 |
| rcarrillocruz | yolanda: mind reviewing ^ pls | 09:19 |
| *** amitgandhinz has joined #openstack-infra | 09:19 | |
| *** rossella_ has quit IRC | 09:19 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Horizon Meeting Chair to new PTL https://review.openstack.org/378417 | 09:19 |
| *** rossella_ has joined #openstack-infra | 09:20 | |
| *** yanyanhu_ has quit IRC | 09:24 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:25 | |
| openstackgerrit | Sam Betts proposed openstack-infra/irc-meetings: Add Networking Cisco IRC Meeting https://review.openstack.org/377622 | 09:26 |
| *** pradiprwt has joined #openstack-infra | 09:27 | |
| yolanda | sure | 09:27 |
| pradiprwt | Hi AJaeger | 09:28 |
| pradiprwt | Please have a look to this bug review https://review.openstack.org/#/c/377639/ | 09:29 |
| rcarrillocruz | thx | 09:32 |
| *** pgadiya has joined #openstack-infra | 09:32 | |
| *** Julien-z_ has quit IRC | 09:35 | |
| *** tuannguyen has joined #openstack-infra | 09:36 | |
| pradiprwt | Hi rcarrillocruz, Please have a look to this bug review https://review.openstack.org/#/c/377639/ | 09:36 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 09:37 |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate openstackci launcher layouts https://review.openstack.org/378458 | 09:37 |
| *** berendt has joined #openstack-infra | 09:39 | |
| *** electrofelix has joined #openstack-infra | 09:39 | |
| *** amoralej is now known as amoralej|out | 09:40 | |
| *** tuanla has joined #openstack-infra | 09:40 | |
| *** gongysh has quit IRC | 09:40 | |
| *** tuannguyen has quit IRC | 09:41 | |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate computes deployed and reachable from 21 to 30 https://review.openstack.org/371551 | 09:44 |
| *** denisra has joined #openstack-infra | 09:44 | |
| openstackgerrit | Merged openstack-infra/system-config: As initial step to deploy with HA, start using pacemaker module https://review.openstack.org/335511 | 09:45 |
| *** electrofelix has quit IRC | 09:45 | |
| *** claudiub|2 has quit IRC | 09:45 | |
| *** electrofelix has joined #openstack-infra | 09:46 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Add the ability to configure bridge name https://review.openstack.org/368692 | 09:49 |
| *** amitgandhinz has quit IRC | 09:51 | |
| *** denisra_ has joined #openstack-infra | 09:53 | |
| ralonsoh | Hi. How can I add a PTL to a new group? According to http://docs.openstack.org/infra/manual/creators.html I need to ask you this | 09:54 |
| *** matrohon has quit IRC | 09:56 | |
| *** denisra has quit IRC | 09:56 | |
| *** ralonsoh has left #openstack-infra | 09:59 | |
| *** ralonsoh has joined #openstack-infra | 09:59 | |
| *** M-docaedo_vector has joined #openstack-infra | 10:01 | |
| *** zhurong has quit IRC | 10:02 | |
| dims | AJaeger : we should ping harlowja and let him know. i'll try to remember to do that today | 10:07 |
| AJaeger | Guest75624: see http://docs.openstack.org/infra/manual/developers.html#automated-testing | 10:09 |
| AJaeger | ralonsoh: tell us the review that merged and your email and I hope somebody from the team will read backscroll and add you. | 10:10 |
| AJaeger | thanks, dims | 10:11 |
| ralonsoh | AJaeger: thank you. https://review.openstack.org/#/c/377794/. Email: rodolfo.alonso.hernandez@intel.com | 10:11 |
| *** kaisers has joined #openstack-infra | 10:12 | |
| *** kaisers_ has joined #openstack-infra | 10:12 | |
| AJaeger | infra-root, please setup the groups for ralonsoh ^ | 10:12 |
| dims | Any git admins around? please nuke http://git.openstack.org/cgit/openstack/tacker-horizon/log/?h=stable/newton as i had a typo in my command line and ended up creating tag from an old SHA | 10:13 |
| *** yamamoto has quit IRC | 10:13 | |
| *** kaisers__ has joined #openstack-infra | 10:14 | |
| *** kaisers_ has quit IRC | 10:14 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add mirror to chocolate openstackci cloud https://review.openstack.org/378499 | 10:15 |
| *** kaisers has quit IRC | 10:16 | |
| *** prometheanfire has quit IRC | 10:17 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add jobs to networking-cisco for testing version compat https://review.openstack.org/378502 | 10:17 |
| *** esikachev has quit IRC | 10:18 | |
| *** prometheanfire has joined #openstack-infra | 10:19 | |
| *** sshnaidm is now known as sshnaidm|lnch | 10:21 | |
| *** ihrachys has quit IRC | 10:22 | |
| lennyb | wznoinsk, hi, faced same failures with devstack commit 69700227a9bdc65acd3aa8798e4eda7e8264dbb5. do you have any understandings of the issue? | 10:23 |
| yolanda | AJaeger, ralonsoh , let me check | 10:23 |
| openstackgerrit | Yuval Brik proposed openstack-infra/system-config: Rename Smaug to Karbor https://review.openstack.org/353309 | 10:24 |
| wznoinsk | lennyb: hi, I guess depends what error youre seeing | 10:24 |
| wznoinsk | lennyb: in my case the subnetpool IP addresses that are picked up sometimes clash with the subnet our external DNS is being configured in /etc/unbound/* inside the build VMs, subnetpool adds a route in the systems and connection to dns drops | 10:25 |
| yolanda | done | 10:25 |
| lennyb | wznoinsk, :), I see that my servers is disconnected from the net during ./stack.sh ( replace ips ... ) and fails since it can't download required code | 10:25 |
| wznoinsk | so probably similar story to mine lennyb, the route inside a VM added for subnetpool may be clashing with address you're using in your CI (public IPs of VMs?) | 10:26 |
| AJaeger | yolanda: do you have any idea why these puppet tests in https://review.openstack.org/355588 fail? Is that a CI problem? | 10:27 |
| yolanda | looking | 10:28 |
| lennyb | wznoinsk, I dont use vm, it's a physical server. I am checking how to configure properly/disable subnet pools | 10:28 |
| yolanda | lint is a legit failure | 10:28 |
| yolanda | 2016-09-27 07:30:35.471500 | manifests/bot.pp - WARNING: class included by relative name on line 8 | 10:28 |
| yolanda | trusty looks as a temporary failure, will need recheck | 10:29 |
| rcarrillocruz | pabelanger: would be good if we can get together and test https://review.openstack.org/#/c/373346/ | 10:30 |
| rcarrillocruz | cos really, i lost like 20min getting auth errors when using openstackci user to do anything | 10:30 |
| *** achevychalov has quit IRC | 10:31 | |
| wznoinsk | lennyb: https://github.com/openstack-dev/devstack/blob/69700227a9bdc65acd3aa8798e4eda7e8264dbb5/lib/neutron_plugins/services/l3#L85 here's the answer | 10:31 |
| openstackgerrit | Merged openstack-infra/system-config: Add mirror to chocolate openstackci cloud https://review.openstack.org/378499 | 10:31 |
| yolanda | AJaeger, so no, trusty is also a legit failure, i got confused by the warnings about ec2 | 10:31 |
| AJaeger | yolanda: how do I replace "include gerritbot" ? | 10:31 |
| rcarrillocruz | turns out when you create the user with openstack client it won't add the user as member of the default project | 10:31 |
| yolanda | real failure is | 10:31 |
| yolanda | 2016-09-27 07:22:14.446254 | Error: Invalid parameter nick on Class[Gerritbot] at /home/jenkins/workspace/gate-infra-puppet-apply-ubuntu-trusty/openstack-infra/system-config/modules/openstack_project/manifests/review.pp:318 on node ubuntu-trusty-osic-cloud1-4559008.openstack.org | 10:31 |
| yolanda | 2016-09-27 07:22:14.446316 | Error: Invalid parameter nick on Class[Gerritbot] at /home/jenkins/workspace/gate-infra-puppet-apply-ubuntu-trusty/openstack-infra/system-config/modules/openstack_project/manifests/review.pp:318 on node ubuntu-trusty-osic-cloud1-4559008.openstack.org | 10:31 |
| *** john-davidge has joined #openstack-infra | 10:31 | |
| yolanda | AJaeger, do include ::gerritbot | 10:31 |
| rcarrillocruz | i.e. the 'Member' role has to be explicitly created and assigned | 10:31 |
| rcarrillocruz | /sadness | 10:31 |
| AJaeger | thanks, yolanda | 10:31 |
| *** john-davidge has left #openstack-infra | 10:32 | |
| lennyb | wznoinsk, so setting SUBNETPOOL_PREFIX_V4=192.168.0.0 should solve the issue | 10:32 |
| openstackgerrit | Isaac Beckman proposed openstack/diskimage-builder: Enable ssh password authentication https://review.openstack.org/376572 | 10:33 |
| *** senk_ has quit IRC | 10:34 | |
| *** senk_ has joined #openstack-infra | 10:35 | |
| *** tuanla has quit IRC | 10:39 | |
| *** salv-orlando has joined #openstack-infra | 10:40 | |
| *** salv-orl_ has quit IRC | 10:40 | |
| *** _degorenko is now known as degorenko | 10:46 | |
| *** amitgandhinz has joined #openstack-infra | 10:47 | |
| *** links has quit IRC | 10:53 | |
| *** priteau has quit IRC | 10:53 | |
| wznoinsk | lennyb: yes, as long as you don't have anything inside your build environment that uses an IP from that subnet (like an external DNS server) | 10:57 |
| wznoinsk | or proxy | 10:58 |
| openstackgerrit | Markos Chandras proposed openstack-infra/glean: Fix SUSE based network configuration https://review.openstack.org/378532 | 11:00 |
| *** dprince has joined #openstack-infra | 11:02 | |
| *** esikachev has joined #openstack-infra | 11:02 | |
| *** links has joined #openstack-infra | 11:02 | |
| openstackgerrit | Bilal Baqar proposed openstack-infra/project-config: Add Juju Charms for PLUMgrid https://review.openstack.org/378534 | 11:02 |
| *** pgadiya has quit IRC | 11:06 | |
| *** sshnaidm|lnch is now known as sshnaidm | 11:08 | |
| *** senk_ has quit IRC | 11:09 | |
| *** sdague has joined #openstack-infra | 11:09 | |
| rcarrillocruz | infra-root , we haz chocolate mirror http://mirror.regionone.infracloud-chocolate.openstack.org/ | 11:13 |
| *** jkilpatr has joined #openstack-infra | 11:14 | |
| *** senk_ has joined #openstack-infra | 11:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Remove the mirror server resource from infracloud clouds https://review.openstack.org/378540 | 11:15 |
| openstackgerrit | Markos Chandras proposed openstack-infra/glean: Fix SUSE based network configuration https://review.openstack.org/378532 | 11:15 |
| *** rtheis has joined #openstack-infra | 11:16 | |
| *** kaisers__ has quit IRC | 11:17 | |
| *** senk_ has quit IRC | 11:19 | |
| *** pgadiya has joined #openstack-infra | 11:19 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add resources to chocolate openstackzuul cloud https://review.openstack.org/378542 | 11:20 |
| *** ssancheztrujillo has joined #openstack-infra | 11:21 | |
| openstackgerrit | Markos Chandras proposed openstack-infra/glean: Fix SUSE based network configuration https://review.openstack.org/378532 | 11:21 |
| *** amitgandhinz has quit IRC | 11:22 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate openstackzuul oscc to Puppetmaster https://review.openstack.org/378545 | 11:23 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate openstackci oscc to Puppetmaster all-clouds https://review.openstack.org/378546 | 11:25 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add resources to chocolate openstackzuul cloud https://review.openstack.org/378542 | 11:28 |
| *** lucas-afk is now known as lucasagomes | 11:29 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Networking Cisco IRC Meeting https://review.openstack.org/377622 | 11:30 |
| *** tuannguyen has joined #openstack-infra | 11:35 | |
| *** ccamacho is now known as ccamacho|lunch | 11:36 | |
| rcarrillocruz | pabelanger: we'll also need https://review.openstack.org/#/c/378552/1 | 11:36 |
| *** pilgrimstack has quit IRC | 11:36 | |
| rcarrillocruz | os_keystone_role => create an openstack role | 11:36 |
| *** pilgrimstack1 has joined #openstack-infra | 11:37 | |
| rcarrillocruz | os_user_role => associates a role between a user and a project | 11:37 |
| *** senk_ has joined #openstack-infra | 11:39 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add jobs to networking-cisco for testing version compat https://review.openstack.org/378502 | 11:39 |
| *** pilgrimstack has joined #openstack-infra | 11:40 | |
| *** pilgrimstack1 has quit IRC | 11:40 | |
| *** zhurong has joined #openstack-infra | 11:41 | |
| *** ldnunes has joined #openstack-infra | 11:42 | |
| *** pilgrimstack has quit IRC | 11:42 | |
| *** tuannguyen has quit IRC | 11:42 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add xenial tempest jobs https://review.openstack.org/377360 | 11:43 |
| *** tuannguyen has joined #openstack-infra | 11:43 | |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate openstackzuul oscc to Puppetmaster https://review.openstack.org/378545 | 11:43 |
| *** salv-orlando has quit IRC | 11:44 | |
| *** EricGonczer_ has joined #openstack-infra | 11:45 | |
| *** ianychoi_ has joined #openstack-infra | 11:46 | |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate openstackci oscc to Puppetmaster all-clouds https://review.openstack.org/378546 | 11:47 |
| openstackgerrit | Merged openstack-infra/system-config: Add resources to chocolate openstackzuul cloud https://review.openstack.org/378542 | 11:47 |
| *** tuannguyen has quit IRC | 11:48 | |
| *** thorst has joined #openstack-infra | 11:48 | |
| *** ianychoi has quit IRC | 11:49 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Remove the mirror server resource from infracloud clouds https://review.openstack.org/378540 | 11:49 |
| *** derekh has quit IRC | 11:52 | |
| *** lezbar has quit IRC | 11:53 | |
| *** markvoelker has quit IRC | 11:53 | |
| *** lezbar has joined #openstack-infra | 11:53 | |
| *** berendt has quit IRC | 11:54 | |
| *** muawiakh_ has joined #openstack-infra | 11:55 | |
| *** muawiak__ has joined #openstack-infra | 11:55 | |
| *** muawiakhan has quit IRC | 11:56 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Add chocolate provider to Nodepool https://review.openstack.org/378559 | 11:56 |
| *** Ravikiran_K has quit IRC | 11:57 | |
| *** danielitit has quit IRC | 11:58 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate oscc cloud to Nodepool https://review.openstack.org/378561 | 11:58 |
| *** baoli has joined #openstack-infra | 11:58 | |
| *** muawiakh_ has quit IRC | 11:59 | |
| *** psilvad has joined #openstack-infra | 12:00 | |
| *** lucasagomes is now known as lucas-bbl | 12:01 | |
| *** sdake has joined #openstack-infra | 12:02 | |
| ttx | Gerrit is slow from here, anyone else noticing that ? | 12:04 |
| *** rfolco_ has joined #openstack-infra | 12:04 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate params and resources to Nodepool node https://review.openstack.org/378564 | 12:05 |
| *** psilvad has quit IRC | 12:06 | |
| vponomaryov | ttx: yes, 502 from time to time | 12:06 |
| EmilienM | ttx: it's also slow here (Canada) | 12:08 |
| openstackgerrit | Merged openstack-infra/system-config: Remove the mirror server resource from infracloud clouds https://review.openstack.org/378540 | 12:08 |
| sdague | ttx: yep | 12:09 |
| *** edmondsw has joined #openstack-infra | 12:09 | |
| dulek | ttx: Slow for me too (EU). | 12:12 |
| *** jpena is now known as jpena|lunch | 12:12 | |
| *** mtanino has joined #openstack-infra | 12:12 | |
| *** gordc has joined #openstack-infra | 12:12 | |
| ianychoi_ | Slow mee too | 12:12 |
| ianychoi_ | s/mee/me | 12:12 |
| *** ianychoi_ is now known as ianychoi | 12:12 | |
| *** amitgandhinz has joined #openstack-infra | 12:13 | |
| therve | sdague, Thanks for you response on the infra ML | 12:14 |
| therve | sdague, 1) I asked for the image mirror. It was discussed during the meeting yesterday | 12:15 |
| therve | sdague, 2) I know that downloading stuff isn't great, but it didn't use to be that bad | 12:15 |
| therve | (3) I'm not subscribed to that list, sorry...) | 12:15 |
| *** baoli has quit IRC | 12:16 | |
| *** psilvad has joined #openstack-infra | 12:16 | |
| sdague | therve: yeh, I remember seeing similar issues with fedora mirrors a couple of years ago. I don't have much insight into how they are managed or monitored | 12:16 |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** mat128 is now known as mat128|afk | 12:17 | |
| therve | sdague, As much as I'd like to blame them, it does sound like an openstack issue though :) | 12:17 |
| mordred | yah ... I think having an image mirror is going to be a good win | 12:17 |
| sdague | therve: could be, realize osic is 2/3 of our nodes | 12:18 |
| therve | Unless we got banned for too much download or something | 12:18 |
| therve | sdague, I was wondering, that's a good info, thanks | 12:18 |
| mordred | yah. I mean, that's why mirrors of things we use is so nice - it's also friendly to upstreams - we download things a LOT :) | 12:18 |
| sdague | so only seeing it on osic is a pretty high statistical probability even if it's across the board issue | 12:18 |
| ralonsoh | yolanda: Thank you! | 12:19 |
| therve | Yeah, sorry we let that issue lingering for so long | 12:19 |
| dmellado | 502 error and slow here too | 12:20 |
| *** pilgrimstack has joined #openstack-infra | 12:21 | |
| *** vikrant has quit IRC | 12:22 | |
| openstackgerrit | Bilal Baqar proposed openstack-infra/project-config: Add Juju Charms for PLUMgrid https://review.openstack.org/378534 | 12:23 |
| ttx | infra-core: gerrit might need a restart to free up memory | 12:24 |
| rcarrillocruz | hey | 12:24 |
| rcarrillocruz | yup, give me a sec | 12:24 |
| *** amitgandhinz has quit IRC | 12:24 | |
| *** yamamoto has quit IRC | 12:25 | |
| *** rhallisey has joined #openstack-infra | 12:26 | |
| rcarrillocruz | done | 12:26 |
| ianychoi | rcarrillocruz, thanks! Seems to be much faster :) | 12:27 |
| rcarrillocruz | yeah, top said it was using max memory | 12:27 |
| *** tuannguyen has joined #openstack-infra | 12:28 | |
| *** jamielennox|away is now known as jamielennox | 12:29 | |
| *** rlu has joined #openstack-infra | 12:30 | |
| *** narayrak has joined #openstack-infra | 12:30 | |
| *** bhavik1 has joined #openstack-infra | 12:31 | |
| *** bhavik1 has quit IRC | 12:31 | |
| *** tobasco has joined #openstack-infra | 12:32 | |
| *** salv-orlando has joined #openstack-infra | 12:33 | |
| *** bhavik1 has joined #openstack-infra | 12:33 | |
| *** amitgandhinz has joined #openstack-infra | 12:33 | |
| *** oanson has quit IRC | 12:34 | |
| *** Julien-zte has joined #openstack-infra | 12:37 | |
| *** mriedem has joined #openstack-infra | 12:37 | |
| *** e0ne has quit IRC | 12:40 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/puppet-apps_site: Glare support for app-catalog https://review.openstack.org/359029 | 12:40 |
| *** vponomaryov has quit IRC | 12:41 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 12:42 |
| *** pzhurba has quit IRC | 12:42 | |
| *** aarefiev has quit IRC | 12:42 | |
| *** pzhurba has joined #openstack-infra | 12:43 | |
| *** e0ne has joined #openstack-infra | 12:43 | |
| *** vponomaryov has joined #openstack-infra | 12:43 | |
| *** rlandy has joined #openstack-infra | 12:44 | |
| openstackgerrit | Andrew Laski proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook https://review.openstack.org/377113 | 12:44 |
| *** aarefiev has joined #openstack-infra | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:45 | |
| *** amitgandhinz has quit IRC | 12:45 | |
| *** yamamoto has quit IRC | 12:45 | |
| *** amitgandhinz has joined #openstack-infra | 12:45 | |
| *** amoralej|out is now known as amoralej | 12:46 | |
| *** rodrigods has quit IRC | 12:47 | |
| *** hashar is now known as hasharAway | 12:47 | |
| *** rodrigods has joined #openstack-infra | 12:47 | |
| *** trown|outtypewww is now known as trown | 12:48 | |
| kfox1111 | anyone have a good example of running a gate job that needs docker with centos7? | 12:48 |
| kfox1111 | I'm running into the cloudinit+docker boot ordering issue I think. | 12:48 |
| mordred | kragniz: our gate nodes shouldn't have cloud-init on them | 12:49 |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| mordred | gah. sorry kragniz | 12:49 |
| kragniz | hmm? | 12:49 |
| mordred | kfox1111: ^^ | 12:49 |
| kragniz | ah :) | 12:49 |
| kfox1111 | oh... so maybe a different issue then. | 12:49 |
| kfox1111 | http://logs.openstack.org/56/378056/7/experimental/gate-kolla-kubernetes-deploy-centos-binary-nv/b440a10/console.html | 12:49 |
| kfox1111 | the sudo systemctl start docker runs but docker never starts. | 12:50 |
| openstackgerrit | Andrew Laski proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook https://review.openstack.org/377113 | 12:50 |
| mordred | oh - boo. that job does not collect things like syslog and whatnot | 12:51 |
| kfox1111 | just starting to write it. still figuring out what to do. | 12:52 |
| kfox1111 | its my first. | 12:52 |
| mordred | kfox1111: so - there is a publisher macro called "devstack-logs" - ignore the name, it's not strictly for devstack | 12:53 |
| mordred | kfox1111: what it'll do is grab anything that's in the "logs" dir and publish it to logs.o.o with the console log | 12:53 |
| mordred | kfox1111: at the end of devstack-gate, we have some stuff that copies relevant log files into that logs dir | 12:54 |
| mordred | at the moment, that's not broken out for ease of reconsumption - although we just specced out doing that | 12:55 |
| *** markvoelker has joined #openstack-infra | 12:55 | |
| * mordred is thinking through easiest way for you to add something similar for your job so that you can debug what's going on here ... | 12:55 | |
| *** EricGonc_ has joined #openstack-infra | 12:56 | |
| kfox1111 | yeah. that sounds good. | 12:56 |
| mordred | yah - I think it's devstack itself that copies syslog and other system logs into the logs dir | 12:57 |
| *** Goneri has joined #openstack-infra | 12:57 | |
| *** david-lyle has joined #openstack-infra | 12:57 | |
| *** kaisers has joined #openstack-infra | 12:58 | |
| *** tuannguyen has quit IRC | 12:58 | |
| *** EricGonczer_ has quit IRC | 12:58 | |
| *** tuannguyen has joined #openstack-infra | 12:58 | |
| *** drawsmcgraw has quit IRC | 12:59 | |
| *** drawsmcgraw has joined #openstack-infra | 12:59 | |
| mordred | kfox1111: ah - you do have the devstack-logs macro ... so all you need to do is add something to the end of your tox target that will copy logs into a dir named logs ... and make sure it does it whether things succeed or fail | 12:59 |
| *** baoli has joined #openstack-infra | 12:59 | |
| mordred | (you may want to make that only do so if a variable is set or something - not sure if devs run the tox target locally or not - if they do, it might be a really annoying addition :) ) | 12:59 |
| kfox1111 | `pwd`/logs? ~/logs? other? | 12:59 |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Add Flag module v3 commons refactor patch https://review.openstack.org/378596 | 13:00 |
| *** tuannguyen has quit IRC | 13:00 | |
| *** tuannguyen has joined #openstack-infra | 13:00 | |
| *** baoli_ has joined #openstack-infra | 13:02 | |
| *** Julien-zte has quit IRC | 13:04 | |
| *** lucas-bbl is now known as lucasagomes | 13:05 | |
| *** baoli has quit IRC | 13:06 | |
| *** yamamoto has quit IRC | 13:07 | |
| kfox1111 | mordred: do you have an example of some code that copies in the logs? | 13:07 |
| *** ccamacho|lunch is now known as ccamacho | 13:08 | |
| *** jamesdenton has joined #openstack-infra | 13:08 | |
| *** Julien-zte has joined #openstack-infra | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** mtanino has quit IRC | 13:09 | |
| *** sshnaidm is now known as sshnaidm|afk | 13:10 | |
| mordred | kfox1111: effectively `pwd`/logs from the dir tox was executed from | 13:11 |
| *** claudiub has joined #openstack-infra | 13:12 | |
| *** psachin has quit IRC | 13:12 | |
| mordred | kfox1111: $WORKSPACE/logs actually - although it's the same thing | 13:13 |
| *** woodster_ has joined #openstack-infra | 13:13 | |
| *** matt-borland has joined #openstack-infra | 13:14 | |
| kfox1111 | perfect. thanks. :) | 13:14 |
| *** eantyshev has left #openstack-infra | 13:14 | |
| *** senk_ has quit IRC | 13:16 | |
| *** sshnaidm|afk has quit IRC | 13:16 | |
| *** senk_ has joined #openstack-infra | 13:16 | |
| *** berendt has joined #openstack-infra | 13:17 | |
| *** berendt has quit IRC | 13:17 | |
| *** kgiusti has joined #openstack-infra | 13:18 | |
| *** jkilpatr has quit IRC | 13:18 | |
| *** pzhurba has quit IRC | 13:18 | |
| *** jkilpatr has joined #openstack-infra | 13:18 | |
| *** rossella_ has quit IRC | 13:19 | |
| *** psachin has joined #openstack-infra | 13:19 | |
| *** rossella_ has joined #openstack-infra | 13:19 | |
| *** jaosorior has quit IRC | 13:20 | |
| pleia2 | fyi, gravestone unveiling and assorted related family things today, not around much | 13:20 |
| *** jaosorior has joined #openstack-infra | 13:21 | |
| *** salv-orlando has quit IRC | 13:21 | |
| *** jpena|lunch is now known as jpena | 13:21 | |
| AJaeger | clarkb: I'm happy now with vponomaryov's manila newton change - do you want to review it as well? https://review.openstack.org/#/c/377360 | 13:23 |
| *** Julien-zte has quit IRC | 13:23 | |
| *** Julien-zte has joined #openstack-infra | 13:24 | |
| *** mat128|afk is now known as mat128 | 13:27 | |
| *** esberglu has joined #openstack-infra | 13:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use command module instead of zuul_runner https://review.openstack.org/369604 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Rename callback_plugins dir to callback https://review.openstack.org/371887 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use environment task parameter instead of parameters https://review.openstack.org/369602 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Rename zuul_runner to command https://review.openstack.org/369603 | 13:30 |
| *** bhavik1 has quit IRC | 13:32 | |
| HeOS | Hello, infra team! If you have no objections, could you please merge this request: https://review.openstack.org/#/c/377639/ ? | 13:33 |
| *** cardeois has joined #openstack-infra | 13:33 | |
| *** pgadiya has quit IRC | 13:34 | |
| *** cardeois has quit IRC | 13:36 | |
| *** esberglu has quit IRC | 13:36 | |
| *** esberglu has joined #openstack-infra | 13:36 | |
| *** tongli has joined #openstack-infra | 13:37 | |
| kfox1111 | mordred: ok, tried that: | 13:38 |
| kfox1111 | https://review.openstack.org/#/c/378056/10 | 13:38 |
| kfox1111 | job failed with POST_FAILURE in 3m 25s (non-voting) now. | 13:38 |
| kfox1111 | never seen that before. | 13:38 |
| *** links has quit IRC | 13:38 | |
| kfox1111 | logs: http://logs.openstack.org/56/378056/10/experimental/gate-kolla-kubernetes-deploy-centos-binary-nv/7f56fb9/console.html | 13:38 |
| *** tonytan4ever has joined #openstack-infra | 13:40 | |
| mordred | kfox1111: wow... this is a fun one :) | 13:40 |
| kfox1111 | :) | 13:40 |
| mordred | oh! you know ... this has nothing to do with the gate ... | 13:41 |
| mordred | but I remember in my brainhole something wonky with docker and fedora/centos as it relates to selinux | 13:41 |
| mordred | which causes it to not be able to start | 13:41 |
| kfox1111 | is selinux enabled on the boxes by default? | 13:41 |
| *** cardeois has joined #openstack-infra | 13:41 | |
| *** esberglu has quit IRC | 13:42 | |
| pabelanger | greghaynes: thanks! | 13:42 |
| mordred | I believe so .. pabelanger you know? | 13:42 |
| kfox1111 | k. I'll add a disable in there... | 13:42 |
| fungi | therve: sheer speculation, but if it's ipv4 then maybe we're overloading their pat (since that environment only has ipv6 for direct global connectivity) | 13:42 |
| fungi | dims: still need anything done with the tacker-horizon repo? | 13:43 |
| *** roxanaghe has joined #openstack-infra | 13:43 | |
| *** eharney has joined #openstack-infra | 13:43 | |
| *** esberglu has joined #openstack-infra | 13:43 | |
| *** roxanaghe has quit IRC | 13:45 | |
| fungi | rcarrillocruz: did you still want to work on the replacement cacti now that we have stats suggesting 4gb is probably sufficient? (after upping the connection table max and then looking at the system load and swap utilization, i' don't think a 2gb replacement will give us enough headroom) | 13:45 |
| mordred | kfox1111: https://github.com/docker/docker/issues/15498 other people having problems running docker on centos-7 in vms | 13:45 |
| pabelanger | mordred: kfox1111: yes, selinux is enabled on centos-7 nodes | 13:45 |
| mordred | kfox1111: I have not read the whole issue yet - it's long - but it does have a bunch of debugging ideas | 13:45 |
| kfox1111 | k. I'll ahve a look through. | 13:46 |
| kfox1111 | I use it in vm's all the time on centos7 in our production systems. | 13:46 |
| kfox1111 | which was why I was asking about cloud init. | 13:46 |
| fungi | rcarrillocruz: the gerrit jvm will use pretty much all of the system memory on review.o.o because it's sized to do so. performance will be determined by resource consumption _within_ the jvm, and for that you're pretty much stuck looking at the javamelody graphs for it | 13:46 |
| mordred | kfox1111: nod. well, I'm curious to know what the issue is :) | 13:47 |
| kfox1111 | there's one issue where cloud-init happens before docker-storage-setup happens, os you cant do docker commands winthin cloud-init scripts. | 13:47 |
| AJaeger | fungi, could you put https://review.openstack.org/376573 on your review stack, please? That's a fix for the zuul-cloner change that broke all jobs on Monday... | 13:47 |
| *** xyang1 has joined #openstack-infra | 13:47 | |
| kfox1111 | brb | 13:47 |
| fungi | AJaeger: lgtm, and great catch | 13:49 |
| *** liusheng has quit IRC | 13:49 | |
| fungi | AJaeger: as for changing the remaining jobs back to using that, we should probably wait a week. i'd rather avoid destabilizing things unnecessarily between now and release day just for the sake of consistency/cleanup | 13:50 |
| AJaeger | thanks, mordred and fungi for simultaneous +2s. I'll +A myself then ;) | 13:50 |
| mordred | ++ | 13:50 |
| openstackgerrit | Zhenguo Niu proposed openstack-infra/puppet-gerritbot: Refactor bot into defined types to allow multiple bots https://review.openstack.org/355588 | 13:50 |
| AJaeger | fungi: YEs, definitely - and perhaps not everything at once ;) | 13:50 |
| *** liusheng has joined #openstack-infra | 13:52 | |
| *** ddieterly has joined #openstack-infra | 13:52 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook https://review.openstack.org/377113 | 13:52 |
| rcarrillocruz | fungi: sure, i can work on that | 13:52 |
| fungi | rcarrillocruz: remember we'll want to copy over the rrd files from the old server to the new one so we don't lose history | 13:53 |
| rcarrillocruz | is there anything wrong with the openstack_infra helper thing? | 13:53 |
| fungi | what helper thing? | 13:53 |
| rcarrillocruz | sec | 13:53 |
| rcarrillocruz | http://logs.openstack.org/64/378564/1/check/gate-openstackci-beaker-centos-7/cfd6901/console.html | 13:53 |
| *** dprince has quit IRC | 13:53 | |
| rcarrillocruz | getting consistent failures on beaker tests | 13:53 |
| fungi | oh, the gem | 13:53 |
| rcarrillocruz | for some reason bundler rspec fails | 13:54 |
| rcarrillocruz | i've checeked the repo and it doesn't have changes lately | 13:54 |
| fungi | LoadError: cannot load such file -- serverspec | 13:54 |
| rcarrillocruz | not sure what's up | 13:54 |
| *** dprince has joined #openstack-infra | 13:54 | |
| fungi | new bundler release? | 13:54 |
| rcarrillocruz | let me see | 13:55 |
| *** Guest92 has joined #openstack-infra | 13:55 | |
| kfox1111 | mordred: docker-storage-setup[2649]: INFO: Volume group backing root filesystem could not be determined | 13:55 |
| kfox1111 | docker-storage-setup[2649]: INFO: Volume group backing root filesystem could not be determined | 13:55 |
| kfox1111 | docker-storage-setup[2649]: ERROR: No valid volume group found. Exiting. | 13:55 |
| kfox1111 | for some reason its not doing the loopback setup. | 13:55 |
| kfox1111 | does the vm setup do any config of docker? cause that seems to be different then the distro defaults. | 13:56 |
| rcarrillocruz | last release is from 15 days ago | 13:56 |
| * rcarrillocruz scratches his head | 13:56 | |
| fungi | rcarrillocruz: is it possible my change to un-pin bundler just merged? | 13:57 |
| rcarrillocruz | also, fungi , pabelanger : we have chocolate mirror online | 13:57 |
| mordred | kfox1111: nope, no config of docker that I'm aware of | 13:57 |
| kfox1111 | thats really strange.... | 13:57 |
| *** fguillot has joined #openstack-infra | 13:57 | |
| rcarrillocruz | fungi: which change, you have it handy? | 13:58 |
| fungi | rcarrillocruz: no such luck, merged 12 days ago https://review.openstack.org/371067 | 13:58 |
| rcarrillocruz | Date: Fri Sep 16 02:11:24 2016 +0000 | 13:58 |
| rcarrillocruz | yeah | 13:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add osic-cloud1-(disk|s3500|3700) to grafana https://review.openstack.org/378642 | 13:59 |
| *** mtanino has joined #openstack-infra | 13:59 | |
| *** hrybacki|afk is now known as hrybacki | 13:59 | |
| pabelanger | rcarrillocruz: great | 14:00 |
| mordred | kfox1111: https://bugzilla.redhat.com/show_bug.cgi?id=1290283 see comment 6 | 14:00 |
| openstack | bugzilla.redhat.com bug 1290283 in docker "Docker Storage Setup enters failed state, is confused looking for non-existent VG "sda2"" [Unspecified,Closed: insufficient_data] - Assigned to vgoyal | 14:00 |
| mordred | kfox1111: (don't know if it's helpful or not) | 14:00 |
| *** Rockyg has joined #openstack-infra | 14:00 | |
| *** andymaier has joined #openstack-infra | 14:01 | |
| *** isaacb has quit IRC | 14:01 | |
| fungi | rcarrillocruz: the log says we're installing beaker-rspec 2.0.0, but https://rubygems.org/gems/beaker-rspec says 5.6.0 is current and 2.0.0 is from 2013 | 14:01 |
| fungi | i wonder if someone flubbed a dependency in a new release of some other gem, or if latest bundler is still having problems resolving dependency versions correctly | 14:02 |
| kfox1111 | its about someone who customized the docker storage setup. which I don't hink we have, | 14:02 |
| kfox1111 | though its behaving like it has... | 14:02 |
| fungi | rcarrillocruz: any idea when that job started failing? just today? | 14:03 |
| rcarrillocruz | i noticed today yeah... | 14:03 |
| rcarrillocruz | two different changes | 14:03 |
| rcarrillocruz | same beaker job | 14:03 |
| rcarrillocruz | with same error | 14:03 |
| openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/377421 | 14:04 |
| dims | fungi : yes please nuke the stable/newton from tacker-horizon | 14:04 |
| *** mtanino has quit IRC | 14:04 | |
| fungi | rcarrillocruz: i guess compare the bundle install list from a working and failing job to see what's changed (if anything) | 14:05 |
| fungi | dims: i've deleted the stable/newton branch of openstack/tacker-horizon formerly at 865ef341a8380edc243c4c43d9c7943f6c264779 | 14:06 |
| AJaeger | harlowja, http://logs.openstack.org/periodic/periodic-nova-py27-with-oslo-master/32d4c97/console.html is failing every time - looks like the tox.ini generated does not work. Could you check and fix it? Or should we remove it since nobody cares? | 14:07 |
| rcarrillocruz | well | 14:08 |
| *** sshnaidm|afk has joined #openstack-infra | 14:08 | |
| dims | thanks fungi | 14:08 |
| rcarrillocruz | so, from a failing job, the serverspec gem is not shown as installed | 14:08 |
| rcarrillocruz | whereas a good one it does | 14:08 |
| rcarrillocruz | working: http://logs.openstack.org/27/368727/4/check/gate-openstackci-beaker-centos-7/5596fa1/console.html.gz | 14:08 |
| rcarrillocruz | not working: http://logs.openstack.org/64/378564/1/check/gate-openstackci-beaker-centos-7/cfd6901/console.html | 14:08 |
| frickler | rcarrillocruz: fungi: previous successful jobs had beaker-rspec 5.6.0. fwiw I was having issues with rubygems.org with other jobs lately | 14:08 |
| cloudnull | mornings | 14:09 |
| rcarrillocruz | frickler: good, thanks | 14:09 |
| *** xarses has quit IRC | 14:09 | |
| *** adrian_otto has joined #openstack-infra | 14:09 | |
| *** kaisers has quit IRC | 14:09 | |
| AJaeger | project-config cores, please review my changes for the track-upstream problems: https://review.openstack.org/378003 and https://review.openstack.org/378016 . | 14:10 |
| *** gsilvis has quit IRC | 14:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix zuul-release-git-prep-upper-constraints https://review.openstack.org/376573 | 14:13 |
| *** rbrndt has joined #openstack-infra | 14:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 14:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 14:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 14:14 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Added additional (disk|s3500|s3700) to entries https://review.openstack.org/378215 | 14:16 |
| anteaya | AJaeger: i'm foggy on what this is fixing: https://review.openstack.org/#/c/378003/3 | 14:17 |
| AJaeger | clarkb: we check the upstream URL in tools/check_valid_gerrit_projects.py | 14:17 |
| AJaeger | anteaya: It's testing the URLs - we didn't do this so far. | 14:17 |
| *** kaisers has joined #openstack-infra | 14:17 | |
| clarkb | AJaeger: oh even better | 14:17 |
| anteaya | can you give me a scenario for why we want this? | 14:17 |
| AJaeger | anteaya: see 378016 for 5 repos that are broken. If we would have 378003 in, it wouldn't have happened. | 14:18 |
| AJaeger | anteaya: let me grab a change for you | 14:18 |
| *** rlu has quit IRC | 14:18 | |
| AJaeger | anteaya: see https://review.openstack.org/373202 - and https://review.openstack.org/377843, https://review.openstack.org/377117 - the last two wouldn't have happened if my change was in | 14:19 |
| AJaeger | clarkb: we can move the change from check_valid_gerrit_projects over to this. I wanted to do smaller steps ;) | 14:19 |
| anteaya | sorry maybe it is because I haven't had any breakfast yet, I still don't follow | 14:19 |
| cloudnull | idk if anyone has a had a chance to look into this but in cloud1 we're seeing irregular error node counts http://grafana.openstack.org/dashboard/db/nodepool-osic?panelId=11&fullscreen And i'm not sure if its something on internal or elsewhere at this point. | 14:20 |
| *** salv-orlando has joined #openstack-infra | 14:20 | |
| AJaeger | anteaya: Currently, if you import a repo with upstream URL, we check the repo. | 14:20 |
| anteaya | last I understood normalize project config removed upstream url lines | 14:20 |
| anteaya | check it for what? | 14:20 |
| AJaeger | anteaya: we check that it's properly setup | 14:20 |
| *** nmagnezi has quit IRC | 14:20 | |
| AJaeger | BUT we have with track-upstream the option to import a repo consistently into our setup. | 14:21 |
| *** raildo has joined #openstack-infra | 14:21 | |
| AJaeger | And those are not removed. | 14:21 |
| AJaeger | And those are not checked yet. | 14:21 |
| anteaya | was this always in place and I have a hole in my memory? | 14:21 |
| AJaeger | anteaya: look at the reviews. When there's track-upstream, then normalize does not remove them. | 14:21 |
| AJaeger | anteaya: yes - only user was openstack-infra/gerrit until two weeks or so ago. | 14:22 |
| AJaeger | Now we have 580+ users - and 7 of those were broken. | 14:22 |
| anteaya | why the increase from gerrit to everyone | 14:22 |
| AJaeger | So, I've written a check that zigo cannot break it... | 14:22 |
| anteaya | oh | 14:23 |
| fungi | anteaya: it was only added for the deb package repos | 14:23 |
| anteaya | okay thanks | 14:23 |
| anteaya | fungi: ah thank you | 14:23 |
| AJaeger | anteaya: that's the current debian package building setup. | 14:23 |
| *** nadya has quit IRC | 14:23 | |
| *** hamzy has quit IRC | 14:23 | |
| fungi | their workflow basically has their packaging repos as forks of other git repositories, and they need to keep those forks up to date | 14:23 |
| *** Julien-zte has quit IRC | 14:24 | |
| anteaya | okay thank you | 14:24 |
| anteaya | track-upstream only tracks repos on our git farm, yes? | 14:24 |
| openstackgerrit | Merged openstack-infra/project-config: Check changed track-upstreams in gerrit/projects.yaml https://review.openstack.org/378003 | 14:24 |
| *** hongbin has joined #openstack-infra | 14:24 | |
| mtreinish | fungi: yeah, responses on our bug. \o/ Despite all our issues with mosquitto the maintainer is at least very active and helpful | 14:24 |
| anteaya | not github repos? | 14:24 |
| fungi | which is similar to how we track upstream gerrit's git repository in our gerrit fork | 14:24 |
| *** Julien-zte has joined #openstack-infra | 14:24 | |
| anteaya | fungi: ah thank you | 14:24 |
| fungi | anteaya: track-upstream can track any upstream git repo. for many of them we're also the upstream, for others we may not be | 14:25 |
| *** xarses has joined #openstack-infra | 14:25 | |
| anteaya | okay thank you | 14:25 |
| *** rlu has joined #openstack-infra | 14:25 | |
| AJaeger | fungi,clarkb: if you have ideas for checks that we should add for track-upstream repos - or for normal import ones - please tell. I can enhance the checks further... | 14:26 |
| fungi | mtreinish: wow, that's a super-helpful triage | 14:26 |
| anteaya | AJaeger: so right now 'properly set up' means the url of the repo is accurate, yes? | 14:26 |
| AJaeger | anteaya: exactly. WE try a git clone on it. So, that's the most basic test. | 14:27 |
| anteaya | okay thank you | 14:27 |
| *** yamahata has joined #openstack-infra | 14:27 | |
| zigo | AJaeger: Thanks for writing this. It was very surprising to me to see that when track-upstream was set, there was no check. I was kind of relying on the checks. If I knew, I would have been a way more careful... | 14:27 |
| *** gsilvis has joined #openstack-infra | 14:28 | |
| *** pfallenop has quit IRC | 14:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix upstream tracking repos https://review.openstack.org/378016 | 14:28 |
| *** lin_yang has joined #openstack-infra | 14:28 | |
| *** darvon has quit IRC | 14:28 | |
| zigo | anteaya: FYI, our workflow is to "git merge -X thiers <FOO>" when upstream release a new release numbered FOO. | 14:28 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 14:29 |
| zigo | anteaya: Then git commit -a --amend to fix the packaging (like dependencies and version in debian/changelog). | 14:29 |
| zigo | anteaya: Then push the whole merge + packaging fix to Gerrit for review. | 14:29 |
| mtreinish | fungi: it does raise a question for the future of how we apply a fix once we get one. Presumably ubuntu will be slow to pull it into the package, if they ever do (older lts's never updated the mosquitto package) | 14:29 |
| zigo | anteaya: To be able to push a merge commit, we need the upstream tag and commits to be there already, otherwise gerrit rejects the "git review". | 14:30 |
| zhenguo | AJaeger: can you please help to approve this https://review.openstack.org/#/c/377268/? thank you | 14:30 |
| zigo | anteaya: If it was accepting it, we wouldn't need the track-upstream... | 14:30 |
| zigo | So maybe there's room for improving Gerrit here. | 14:30 |
| *** baoli_ has quit IRC | 14:31 | |
| EmilienM | fungi, clarkb: good morning - FYI infra beaker jobs might break, see https://review.openstack.org/#/c/378648/ and https://github.com/puppetlabs/beaker-rspec/pull/86 | 14:31 |
| *** baoli has joined #openstack-infra | 14:31 | |
| AJaeger | zhenguo: what is it about? | 14:31 |
| anteaya | EmilienM: thanks my skim of backscroll indicates we have been hitting beaker issues today | 14:32 |
| anteaya | EmilienM: thanks for sharing | 14:32 |
| EmilienM | cool np | 14:32 |
| anteaya | zigo: thanks for the explanation | 14:32 |
| zhenguo | AJaeger: Add a project statusbot and meetbot :) | 14:32 |
| *** muawiak__ has quit IRC | 14:33 | |
| AJaeger | zhenguo: change for system-config? I don't review those... | 14:33 |
| *** lennyb has quit IRC | 14:34 | |
| zhenguo | AJaeger: yes, oh sorry I don't know about that | 14:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin beaker to 2.x releases https://review.openstack.org/378670 | 14:35 |
| EmilienM | anteaya: here's a quick fix ^ | 14:35 |
| anteaya | EmilienM: thank you | 14:35 |
| *** shardy has joined #openstack-infra | 14:36 | |
| fungi | EmilienM: thanks, we had been trying to figure out what was going on there | 14:36 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin beaker to 2.x releases https://review.openstack.org/378670 | 14:36 |
| rcarrillocruz | oh nice | 14:36 |
| *** muawiakhan has joined #openstack-infra | 14:36 | |
| rcarrillocruz | thanks EmilienM , i was getting that on my changes | 14:36 |
| EmilienM | http://logs.openstack.org/29/359029/34/check/gate-openstackci-beaker-centos-7/351d01d/console.html#_2016-09-28_14_01_58_722055 | 14:37 |
| fungi | EmilienM: for some reason it was installing an ancient (2.0.0) beaker-rspec gem rather than the current version | 14:37 |
| EmilienM | I confirm we have the same problem in infra | 14:37 |
| EmilienM | fungi: yes! | 14:37 |
| EmilienM | so the PR is WIP in beaker-rspec | 14:38 |
| EmilienM | I would suggest you to merge my pin until it's solved upstream | 14:38 |
| EmilienM | if you want, i'll keep track on this PR and update both infra & puppet gemspecs when ready | 14:38 |
| *** spzala has joined #openstack-infra | 14:38 | |
| *** baoli has quit IRC | 14:38 | |
| anteaya | EmilienM: you're awesome | 14:38 |
| anteaya | thank you | 14:38 |
| fungi | thanks EmilienM, Hunner! | 14:38 |
| EmilienM | well, trying to unblock CI | 14:38 |
| clarkb | cloudnull: pabelanger I can help take a look at those errors when properly awake | 14:38 |
| anteaya | EmilienM: and awesome | 14:39 |
| *** caowei has quit IRC | 14:39 | |
| *** annegentle has joined #openstack-infra | 14:41 | |
| fungi | clarkb: clarkb: pabelanger: any chance it's related to the reported network connectivity issues for jobs that are trying to download things from the internet at large? | 14:41 |
| fungi | er, cloudnull ^ | 14:41 |
| fungi | clarkb: cloudnull: pabelanger: http://lists.openstack.org/pipermail/openstack-infra/2016-September/004749.html | 14:42 |
| clarkb | my guess is something related to our image shuffling yesterday | 14:43 |
| clarkb | I think your guess about NAT an ddownloads is likely | 14:43 |
| fungi | weird, when did github add gerrit-like inline commenting for code review? | 14:44 |
| pabelanger | clarkb: cloudnull: sorry, which errors? | 14:44 |
| mordred | fungi: a couple of weeks ago | 14:44 |
| mordred | fungi: they also added a "Review" primitive | 14:44 |
| openstackgerrit | Sarafraj Singh proposed openstack-infra/devstack-gate: Do not merge: Patch to test LM in nova https://review.openstack.org/378015 | 14:44 |
| fungi | mordred: scary | 14:44 |
| clarkb | pabelanger: apparently new boot errors in osic | 14:44 |
| mordred | fungi: however, they did not seem to figure out that a reviewer might want to change their vote | 14:44 |
| fungi | hah | 14:44 |
| mordred | fungi: OR that a new patchset might should clear out old votes | 14:44 |
| pabelanger | clarkb: Ya, just seen scrollback | 14:44 |
| *** EricGonc_ has quit IRC | 14:44 | |
| pabelanger | I can look now | 14:44 |
| fungi | clearly they hadn't ever used gerrit | 14:45 |
| *** burgerk has joined #openstack-infra | 14:45 | |
| clarkb | pabelanger: ok my hunch is its related to our image deduping somehow | 14:45 |
| pabelanger | OpenStackCloudException: Error in creating instance (Inner Exception: Can not find requested image (HTTP 400) (Request-ID: req-810b9f94-bf8e-4451-a515-ebce164c9113)) | 14:45 |
| pabelanger | yup | 14:45 |
| mordred | fungi: and also clearly they didn't actually think through the workflow at all, since both of those are fairly obvious things, tbh | 14:45 |
| fungi | mordred: sure. it seems like they implemented something based on a rough and incomplete description someone provided about how code review is typically done outside github | 14:46 |
| *** Na3iL is now known as nzoueidi | 14:47 | |
| pabelanger | clarkb: cloudnull: looks to be an issue with the ubuntu-xenial image not being found | 14:48 |
| mordred | fungi: well - you're generous there ... I'm guessing they implemented something without having considered that a world outside of github exists | 14:48 |
| irtermite | pabelanger: yea, I believe cloudnull brought that up late last night and asked if someone was going to upload it | 14:48 |
| clarkb | pabelanger: cloudnull can ptobably just queue a reupload and see if that solves it | 14:48 |
| *** rlu has quit IRC | 14:49 | |
| clarkb | but maybe should consider undeduping if this sort of problem persists | 14:49 |
| pabelanger | I think I see the issue | 14:50 |
| fungi | mordred: i have to assume that their _interest_ in adding code review at all was somehow informed by workflows outside github | 14:50 |
| pabelanger | I think osic-cloud1-s3700 uploaded a never image, which had a different ID | 14:50 |
| *** adrian_otto has quit IRC | 14:50 | |
| pabelanger | http://paste.openstack.org/show/583319/ | 14:51 |
| clarkb | I did restart the builder while s3700 had a few images remaining | 14:51 |
| clarkb | in order to pick up centos7 and f24 fixes | 14:51 |
| pabelanger | so, maybe we delete the xenial images, and force an reupload | 14:51 |
| *** yamamoto has quit IRC | 14:52 | |
| pabelanger | see if nodepool does the right thing | 14:52 |
| *** muawiakhan has quit IRC | 14:52 | |
| *** yamamoto has joined #openstack-infra | 14:52 | |
| clarkb | pabelanger: ok | 14:53 |
| clarkb | I dont think we need to delete first though | 14:53 |
| clarkb | since nodepool.should just use current image | 14:53 |
| *** Swami has joined #openstack-infra | 14:53 | |
| *** kmartin has joined #openstack-infra | 14:54 | |
| anteaya | fungi: clarkb I'm going with logstash will fill up again before the end of today: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=422&rra_id=all | 14:55 |
| *** ddieterly has quit IRC | 14:56 | |
| anteaya | the good news is that the cacti graph doesn't seem to have gaps in it since fungi upped conntrack | 14:56 |
| *** smarcet has joined #openstack-infra | 14:56 | |
| fungi | anteaya: yep, did you see my calculations yesterday? anyway we can go ahead and merge clarkb's change to disable debug logging if nobody has done so yet | 14:56 |
| *** ddieterly has joined #openstack-infra | 14:56 | |
| anteaya | I saw your calculations saying 34 hours from when you cleared it out | 14:56 |
| anteaya | I don't think we will make it to 34 hours | 14:57 |
| fungi | keep in mind that was based on an assumption that the activity volume i observed over a 5-minute period was representative | 14:57 |
| anteaya | oh yes, I'm not saying you are or were in accurate | 14:57 |
| fungi | anyway, i've gone ahead and approved 378104 | 14:57 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper: Pin beaker to 2.x releases https://review.openstack.org/378670 | 14:57 |
| fungi | that should stop it dead in its tracks | 14:57 |
| *** hamzy has joined #openstack-infra | 14:57 | |
| anteaya | I was attempting to say you were being generous | 14:58 |
| anteaya | obviously I missed | 14:58 |
| anteaya | and thank you | 14:58 |
| fungi | oh, yes my "estimate" should have come with an implied +/- 24 hours ;) | 14:58 |
| anteaya | ah | 14:58 |
| *** Guest78091 is now known as redrobot | 14:59 | |
| anteaya | perhaps it did in which case you are bang on | 14:59 |
| fungi | it was really an attempt to gauge the order of magnitude of log volume we were generating on that server | 14:59 |
| anteaya | indeed | 14:59 |
| anteaya | really big is a bit vauge | 14:59 |
| anteaya | vague | 14:59 |
| fungi | so "fill up in a day" vs "in a week" or "month" | 14:59 |
| *** yuval has quit IRC | 14:59 | |
| anteaya | yup | 15:00 |
| *** adrian_otto has joined #openstack-infra | 15:00 | |
| *** armax has joined #openstack-infra | 15:01 | |
| *** zhurong has quit IRC | 15:01 | |
| anteaya | the conntrack increase seems to have mostly worked, here is one with a gap that I believe occured since the change: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=304 | 15:02 |
| *** marst has joined #openstack-infra | 15:02 | |
| *** andreas_s has quit IRC | 15:02 | |
| *** lennyb has joined #openstack-infra | 15:03 | |
| openstackgerrit | Merged openstack-infra/puppet-log_processor: Reduce log client logging by default https://review.openstack.org/378104 | 15:03 |
| *** r-mibu has quit IRC | 15:03 | |
| *** r-mibu has joined #openstack-infra | 15:03 | |
| *** rcernin has quit IRC | 15:04 | |
| anteaya | everything I have clicked on that isn't infra cloud seems to have been addressed by the conntrack increase | 15:04 |
| anteaya | this one is all over the place: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=307 | 15:04 |
| *** nzoueidi has quit IRC | 15:05 | |
| openstackgerrit | Jp Maxwell proposed openstack-infra/system-config: Adding user 'maxwell' to OpenStack ID Production https://review.openstack.org/378686 | 15:05 |
| *** jlanoux has quit IRC | 15:05 | |
| *** nzoueidi has joined #openstack-infra | 15:05 | |
| anteaya | that seems to be par for the course for the infra cloud nodes, very spotty smtp reporting | 15:06 |
| *** Swami_ has joined #openstack-infra | 15:06 | |
| *** tonytan_brb has joined #openstack-infra | 15:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Make sure we're matching image status properly https://review.openstack.org/377164 | 15:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Normalize images https://review.openstack.org/377163 | 15:06 |
| *** caowei has joined #openstack-infra | 15:07 | |
| *** darvon has joined #openstack-infra | 15:07 | |
| *** pfallenop has joined #openstack-infra | 15:07 | |
| *** pfallenop has quit IRC | 15:07 | |
| *** pfallenop has joined #openstack-infra | 15:07 | |
| pabelanger | mordred: I think you were saying, shade will load an image into memory to calculate the md5sum? Is that something we could add to diskimage-builder? When we create the 1160.qcow2 image, also create 1160.qcow2.md5sum, and tell shade to use it? | 15:07 |
| *** tphummel has joined #openstack-infra | 15:08 | |
| mordred | pabelanger: sure. we'll need to calculate the sha256sum too | 15:09 |
| pabelanger | kk | 15:09 |
| *** Swami has quit IRC | 15:09 | |
| mordred | pabelanger: I think we may just want to have that be a nodepool thing though ... create_image takes an md5 and sha256 parameter which will cause it to not calculate those | 15:09 |
| *** tonytan4ever has quit IRC | 15:09 | |
| pabelanger | Ya, that works | 15:09 |
| *** GheRivero has joined #openstack-infra | 15:10 | |
| pabelanger | clarkb: cloudnull: okay, ubuntu-xenial images sync'd. Checking logs now for more failures | 15:10 |
| mordred | pabelanger: so if we teach dib about making files and then teach nodepool to read those files and then pass the contents into the create_image call - I think that's the joy | 15:10 |
| pabelanger | mordred: cool | 15:11 |
| pabelanger | clarkb: I do see some dupe images in osic-cloud1 (eg: nodepool-fedora-24-1148) http://paste.openstack.org/show/583328/ I wonder if we have a race going on | 15:12 |
| *** kzaitsev_mb has quit IRC | 15:13 | |
| *** tdasilva has quit IRC | 15:14 | |
| anteaya | what does the number at the end mean? 1160, 1158, 1148 is that the incremented id of the creation of that image? | 15:14 |
| clarkb | pabelanger: ya I think its a race we dont serialize we will.have those | 15:14 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 15:14 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 15:14 |
| clarkb | pabelanger: I am also a little worried about the corresponding race on the delete side | 15:14 |
| clarkb | pabelanger: eg delete image 123 with uuid foo shared by image 156 | 15:14 |
| pabelanger | anteaya: yes, that is the DIB id of said image | 15:14 |
| anteaya | pabelanger: ah thanks | 15:15 |
| clarkb | now 156 doesnt work if its upload hasnt finished for some reason. except we only delete the previous days so should be fine | 15:15 |
| *** vinaypotluri has joined #openstack-infra | 15:15 | |
| clarkb | I think in practice we willbe ok just need to watch for issues | 15:15 |
| openstackgerrit | Merged openstack-infra/nodepool: Add missing month argument https://review.openstack.org/364151 | 15:16 |
| *** thcipriani|afk is now known as thcipriani | 15:16 | |
| *** amotoki has quit IRC | 15:16 | |
| pabelanger | clarkb: have we considered just adding a image-upload: True / False field for providers? | 15:17 |
| pabelanger | actually, not sure that would fix things either. Since provider would be pinned to a specific image | 15:17 |
| pabelanger | nevermind | 15:17 |
| *** nadya has joined #openstack-infra | 15:18 | |
| clarkb | pabelanger: there are things we could do to only delete the image in glace if its uuid isnt in any other row in thr db | 15:19 |
| clarkb | I will ptobably work on that patch today | 15:19 |
| *** zz_dimtruck is now known as dimtruck | 15:19 | |
| *** caowei has quit IRC | 15:20 | |
| clarkb | I just dont know if it will be an actual issue in practice due to our keep two images policy | 15:20 |
| dirk | can anyone help me with a 2nd pair of eyes on a jenkins job failure? | 15:20 |
| dirk | for some reason I think I don't have python2.7 installed on the ubuntu-xenial image. can that be the case? | 15:20 |
| anteaya | dirk: can you share the url for a log? | 15:21 |
| dirk | http://logs.openstack.org/50/378550/2/check/gate-requirements-tox-py27-with-upper-constraints-ubuntu-xenial/5346028/console.html | 15:21 |
| dirk | anteaya: | 15:21 |
| dirk | anteaya: the error message that caught my eye is | 15:21 |
| *** lucasagomes is now known as lucas-hungry | 15:22 | |
| dirk | /usr/local/jenkins/slave_scripts/run-tox.sh: .tox/py27-with-upper-constraints/bin/pip: /home/jenkins/workspace/gate-requirements-tox-py27-with-upper-constraints-ubu: bad interpreter: Permission denied | 15:22 |
| *** yamahata has quit IRC | 15:22 | |
| anteaya | thanks I'm looking, I see the error, thank you | 15:24 |
| pabelanger | I suspect the shebang is too long (because of job name length) which is causing tox to fail | 15:25 |
| pabelanger | >128 path causes the failure | 15:26 |
| pabelanger | is it possible the job changed recently? | 15:26 |
| *** tuannguyen has quit IRC | 15:26 | |
| *** apuimedo is now known as apuimedo|away | 15:27 | |
| *** rfolco_ has quit IRC | 15:27 | |
| jeblair | clarkb: is it okay for me to stop nodepool, or are we uploading images you need? | 15:27 |
| pabelanger | I have some images I'd like to finish today | 15:28 |
| *** mdrabe has joined #openstack-infra | 15:28 | |
| anteaya | dirk: yes it says it is installing python3, it doesn't say it is installing python2, I had thought python2 came with xenial | 15:29 |
| dirk | pabelanger: it is a recent problem, yes | 15:29 |
| anteaya | dirk: python2 isn't installed by default on xenial: https://wiki.ubuntu.com/XenialXerus/ReleaseNotes/#Python_3 | 15:30 |
| jeblair | pabelanger: can you elaborate? | 15:30 |
| dirk | anteaya: great. so how do I tell $zuul to install it for me? | 15:31 |
| *** baoli has joined #openstack-infra | 15:31 | |
| *** rfolco_ has joined #openstack-infra | 15:31 | |
| dirk | I thought it magically detects that based on the job name | 15:31 |
| pabelanger | jeblair: we are doing builds with diskimage-builder 1.20.0 to include or ssh host key fixes, and new release of glean to fix rax-iad. So, would be nice to let them finish, however they could be pushed back another day or so if needed | 15:31 |
| narayrak | clarkb:rajinir: yesterdays CI issues | 15:32 |
| narayrak | With the introduction of subnetpools, new variable SUBNETPOOL_PREFIX_V4 defaults to 10.0.0.0/8 which interfered our floating IP/network range and created routing issue | 15:32 |
| narayrak | https://github.com/openstack-dev/devstack/commit/69700227a9bdc65acd3aa8798e4eda7e8264dbb5 | 15:32 |
| narayrak | now have to handle SUBNETPOOL_PREFIX_V4, along with FIXED_RANGE | 15:32 |
| narayrak | + local replace_range=${SUBNETPOOL_PREFIX_V4} | 15:32 |
| narayrak | + if [[ -z "${SUBNETPOOL_V4_ID}" ]]; then | 15:32 |
| narayrak | + replace_range=${FIXED_RANGE} | 15:32 |
| narayrak | + fi | 15:32 |
| narayrak | + sudo ip route replace $replace_range via $ROUTER_GW_IP | 15:32 |
| *** rajinir has joined #openstack-infra | 15:32 | |
| anteaya | narayrak: in future please use a paste service | 15:32 |
| *** caowei has joined #openstack-infra | 15:32 | |
| jeblair | pabelanger, fungi: okay, i may need someone else to take over the 'split-up-nodepool' work, or perhaps we should abandon it | 15:33 |
| anteaya | dirk: well just to check pabelanger's theory do you have a log from a passing job? | 15:33 |
| jeblair | i'm in the worst timezone for it | 15:33 |
| narayrak | anteaya:Sure, thanks | 15:33 |
| anteaya | narayrak: thank you | 15:33 |
| jeblair | it was meant to be a quick temporary fix | 15:33 |
| rajinir | narayrak: This fixes the subnet issue with the CI builds https://review.openstack.org/#/c/378063/ | 15:34 |
| dirk | anteaya: sure http://logs.openstack.org/77/339077/3/check/gate-requirements-tox-py27-with-upper-constraints/e620cdf/console.html.gz | 15:34 |
| anteaya | thank you | 15:34 |
| pabelanger | jeblair: sure, I'm happy to wait another day for images to land the nodepool split | 15:34 |
| dirk | anteaya: its not passing but getting beyond that point | 15:34 |
| *** smarcet1981 has joined #openstack-infra | 15:35 | |
| dirk | anteaya: so yes, it got longer due to the ubuntu-xenial suffix | 15:35 |
| *** jkilpatr has quit IRC | 15:35 | |
| jeblair | pabelanger: well, i don't think you should have to wait. it's possible to do both things at once, just not in my timezone. | 15:35 |
| openstackgerrit | Kyle Haley proposed openstack-infra/project-config: Make docs non-voting for quark, remove template https://review.openstack.org/377908 | 15:35 |
| *** ihrachys has joined #openstack-infra | 15:36 | |
| jeblair | electrofelix: can you look at https://review.openstack.org/378046 please? | 15:36 |
| jeblair | zxiiro: ^ | 15:36 |
| anteaya | dirk: so not passing on ubuntu xenial yet? | 15:36 |
| cloudnull | pabelanger: looks like the failure rate is dropping off. | 15:36 |
| *** smarcet has quit IRC | 15:37 | |
| *** andymaier has quit IRC | 15:37 | |
| dirk | anteaya: I think it was running on xenail before | 15:38 |
| *** OferBarber_ has joined #openstack-infra | 15:38 | |
| dirk | just without the -ubuntu-xenial suffix | 15:38 |
| clarkb | jeblair: pabelanger we can always requeue images after any splitting | 15:38 |
| dirk | so this might mean that pabelanger is right | 15:38 |
| *** ihrachys has quit IRC | 15:39 | |
| anteaya | dirk: I'm looking at http://logs.openstack.org/50/378550/2/check/gate-requirements-python27-ubuntu-xenial/def4397/console.html | 15:39 |
| *** ihrachys has joined #openstack-infra | 15:39 | |
| anteaya | a passing job from the same patch and I don't see in the log where python2.7 is installed | 15:40 |
| pabelanger | dirk: it should be easy to reproduce locally, just setup the same path locally, in your /tmp folder, then run tox. You'll see the same bad interperter: Permission denied problem | 15:40 |
| *** dimtruck is now known as zz_dimtruck | 15:40 | |
| *** OferBarber has quit IRC | 15:40 | |
| dirk | anteaya: yep | 15:41 |
| narayrak | rajinir:thank you | 15:41 |
| anteaya | so yeah, right now I'm in agreement with pabelanger's theory | 15:41 |
| anteaya | dirk: let me know how the local test goes | 15:41 |
| kfox1111 | mordred: had a theory... maybe its as simple as docker not giving rights to the jenkins users to do docker ps? :/ | 15:42 |
| pabelanger | clarkb: sure, I'm okay with that | 15:42 |
| kfox1111 | testing now. | 15:42 |
| mordred | kfox1111: this is a good theory - you usually need to be in a special group to run docker commands | 15:43 |
| kfox1111 | yeah. | 15:43 |
| kfox1111 | didnt occur to me until just now. | 15:44 |
| * mordred is sad he didn't think of it | 15:44 | |
| kfox1111 | me neither. andn I use docker a million times a day. :) | 15:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Delete ping test environment in periodic jobs https://review.openstack.org/346134 | 15:44 |
| dirk | pabelanger: anteaya : bingo, yeap, it is the path length | 15:44 |
| *** jlanoux has joined #openstack-infra | 15:44 | |
| anteaya | dirk: thank you | 15:45 |
| anteaya | that might be my first instance of job failing due to name too long | 15:45 |
| openstackgerrit | Kyle Haley proposed openstack-infra/project-config: Make docs non-voting for quark, remove template https://review.openstack.org/377908 | 15:45 |
| electrofelix | jeblair: since JJB patches do get benefit from checking against a more complex set, we might need a replacement for just JJB, maybe just copy the existing into JJB for use as part of functional tests. Fully understand why removing the existing one makes sense for project-config | 15:45 |
| *** Guest75624 is now known as BobBall | 15:45 | |
| dirk | so we have to rename the job? | 15:45 |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config: Add scenarios from Neutron to multinode dvr full job https://review.openstack.org/355344 | 15:45 |
| anteaya | dirk: yes | 15:46 |
| anteaya | apparently 63 characters is too long | 15:46 |
| robcresswell | Could really use a review on https://review.openstack.org/#/c/378406/ if possible; Horizons gate is totally locked up. | 15:46 |
| anteaya | robcresswell: what is going on that making the integration tests not voting is the way forward? | 15:47 |
| *** tiswanso has joined #openstack-infra | 15:47 | |
| *** jkilpatr has joined #openstack-infra | 15:48 | |
| dirk | anteaya: thanks | 15:48 |
| anteaya | dirk: thank you | 15:48 |
| robcresswell | anteaya: Time, primarily. It's either hold everything until we can fix them, or a couple of people work on them and the other reviewers carry on as normal | 15:49 |
| anteaya | but what is the issue? | 15:49 |
| anteaya | what is broken? | 15:49 |
| robcresswell | anteaya: We don't know yet. We've been getting intermittent failures across a broad range of tests | 15:49 |
| anteaya | can you share some logs? | 15:50 |
| anteaya | perhaps more eyes can assist | 15:50 |
| anteaya | I'm not comfortable making integration tests non voting | 15:50 |
| *** mdrabe has quit IRC | 15:51 | |
| jeblair | electrofelix: makes sense. should we proceed with landing the patch to disable while you work on the func testing? or would you like me to make the compare-xml job work (by installing the 'afs' jjb module like we're doing in project-config)? | 15:51 |
| *** mdrabe has joined #openstack-infra | 15:51 | |
| robcresswell | Heh, Horizon PTL doesnt have control over its own tests now? | 15:51 |
| robcresswell | Here is one example http://logs.openstack.org/38/373738/3/gate/gate-horizon-dsvm-integration-deprecated-ubuntu-xenial/faf2992/ | 15:52 |
| anteaya | I'm saying that integration tests affect many projects | 15:52 |
| *** tuannguyen has joined #openstack-infra | 15:52 | |
| anteaya | and that I'm not comfortable just being asked to toggle a switch | 15:52 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config: Rename py27-with-upper-constraints to py27-check-uc https://review.openstack.org/378732 | 15:52 |
| *** tuannguyen has quit IRC | 15:53 | |
| electrofelix | jeblair: yes, I don't see any reason to block, I'll look at what's needed for the func testing tomorrow | 15:53 |
| jeblair | electrofelix: cool, thx | 15:53 |
| dirk | anteaya: pabelanger : your review appreciated : https://review.openstack.org/378732 | 15:53 |
| openstackgerrit | Merged openstack-infra/project-config: Run Nova post_test_hook on placement job https://review.openstack.org/376537 | 15:53 |
| *** tuannguyen has joined #openstack-infra | 15:53 | |
| robcresswell | anteaya: I'm not sure I follow. It only affects the potential stability of Horizon or its plugins, nothing else. | 15:54 |
| electrofelix | jeblair: course the other alternative would be to teach JJB how to pick up extra modules in project-config automagically ;-) | 15:55 |
| electrofelix | but I expect that would be just delaying the inevitable need to remove the compare job anyway | 15:55 |
| anteaya | robcresswell: so horizon integrated tests don't affect any other integrated projects? | 15:55 |
| *** EricGonczer_ has joined #openstack-infra | 15:56 | |
| jeblair | electrofelix: yes.... and yes. :) | 15:56 |
| *** electrofelix has quit IRC | 15:56 | |
| robcresswell | anteaya: No, they just run Horizon with whatever changes against devstack. Its not Tempest | 15:57 |
| robcresswell | Just end to end testing via selenium | 15:57 |
| anteaya | dirk: why not just rename py27 with upper constraints in the tox file rather than adding an additional entry: https://review.openstack.org/#/c/378728/1 | 15:57 |
| zxiiro | electrofelix: waynr I'm at a conference this week so will miss the sprint. I think we still have many patches from the last sprint that needs review anyway. I've already reviewed most of them and can take a look again if necessary between flights. | 15:57 |
| anteaya | robcresswell: oh I beg your pardon then, my understanding of the use of integrated means 'all the projects tested together' | 15:58 |
| *** tuannguyen has quit IRC | 15:58 | |
| dirk | anteaya: hmm, I guess it depends on the merge order.. if the infra change gets merged first, I can just to a review that does the rename | 15:58 |
| dirk | I just added the depends-on as I don't know how likely it is that it would get merged otherwise when the tox.ini entry isn't there | 15:59 |
| anteaya | dirk: ah okay | 15:59 |
| anteaya | fair enough | 15:59 |
| dirk | in the past infra changes were not merged when the tox.ini did not exist properly | 15:59 |
| *** tphummel has quit IRC | 15:59 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on Summit Event times getting local start/end date for summit events was changing the original value instead of clonnig it. https://review.openstack.org/378745 | 15:59 |
| *** tonytan_brb is now known as tonytan4ever | 15:59 | |
| anteaya | dirk: how wise of us | 15:59 |
| *** nadya has quit IRC | 16:00 | |
| dirk | anteaya: yes, I don't argue against that :) | 16:00 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on Summit Event times getting local start/end date for summit events was changing the original value instead of clonnig it. https://review.openstack.org/378745 | 16:00 |
| *** narayrak has quit IRC | 16:00 | |
| *** krtaylor has quit IRC | 16:01 | |
| *** slaweq_ has joined #openstack-infra | 16:01 | |
| *** narayrak has joined #openstack-infra | 16:01 | |
| anteaya | dirk: reviewed, I see a missed bit that needs to be removed | 16:01 |
| *** berendt has joined #openstack-infra | 16:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/377431 | 16:03 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/project-config: Rename py27-with-upper-constraints to py27-check-uc https://review.openstack.org/378732 | 16:03 |
| *** nadya has joined #openstack-infra | 16:03 | |
| *** mtanino has joined #openstack-infra | 16:04 | |
| openstackgerrit | Merged openstack-infra/nodepool: Use sequence znodes for image upload numbers https://review.openstack.org/371045 | 16:04 |
| dirk | anteaya: thx | 16:04 |
| openstackgerrit | Merged openstack-infra/nodepool: Client change to support ZK image build requests https://review.openstack.org/371767 | 16:04 |
| openstackgerrit | Merged openstack-infra/nodepool: Short-circuit builder processing on shutdown https://review.openstack.org/371770 | 16:04 |
| *** caphrim007 has joined #openstack-infra | 16:05 | |
| anteaya | dirk: welcome | 16:05 |
| openstackgerrit | Merged openstack-infra/nodepool: Handle ZooKeeper lost connections https://review.openstack.org/373499 | 16:05 |
| openstackgerrit | Merged openstack-infra/nodepool: Handle config reload race https://review.openstack.org/374414 | 16:05 |
| openstackgerrit | Merged openstack-infra/nodepool: Add UploadWorker skeleton to the builder https://review.openstack.org/377065 | 16:05 |
| *** tuannguyen has joined #openstack-infra | 16:06 | |
| *** gyee has joined #openstack-infra | 16:07 | |
| *** ijw has joined #openstack-infra | 16:07 | |
| *** Julien-zte has quit IRC | 16:08 | |
| robcresswell | thanks for the review anteaya | 16:08 |
| anteaya | you're welcome, thanks for explaining the situation to me, I appreciate it | 16:09 |
| *** rossella_ has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Add JJB AFS module https://review.openstack.org/378045 | 16:09 |
| *** eliqiao has quit IRC | 16:10 | |
| *** tdasilva has joined #openstack-infra | 16:11 | |
| *** thorst_ has joined #openstack-infra | 16:11 | |
| *** roxanaghe has joined #openstack-infra | 16:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Stop running compare-xml jobs https://review.openstack.org/378046 | 16:12 |
| openstackgerrit | Merged openstack-infra/project-config: Publish infra docs to AFS https://review.openstack.org/377930 | 16:12 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix token cache recursion https://review.openstack.org/378756 | 16:12 |
| openstackgerrit | Merged openstack-infra/project-config: Enable layout-checks.py again https://review.openstack.org/378275 | 16:12 |
| waynr | zxiiro: thanks for the heads up | 16:12 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix token cache recursion https://review.openstack.org/378756 | 16:13 |
| *** thorst has quit IRC | 16:13 | |
| pabelanger | dirk: anteaya: I'm going to defer to AJaeger on the upper-constraints job, I know he's been doing a lot of work on it | 16:14 |
| dirk | pabelanger: np | 16:14 |
| *** njohnston has quit IRC | 16:15 | |
| openstackgerrit | Bilal Baqar proposed openstack-infra/project-config: Add Juju Charms for PLUMgrid https://review.openstack.org/378534 | 16:15 |
| *** njohnston has joined #openstack-infra | 16:16 | |
| *** jpich has quit IRC | 16:16 | |
| anteaya | pabelanger: thanks for the pointer to the job name length | 16:16 |
| *** ddieterly is now known as ddieterly[away] | 16:18 | |
| mordred | clarkb: have you seen this: http://logs.openstack.org/49/377049/1/check/gate-dsvm-nodepool-src-shade/e7e180b/logs/screen-nodepool-builder.txt.gz#_2016-09-26_21_12_57_289 ? | 16:19 |
| mordred | clarkb: I'm seeing that on all of the nodepool/shade jobs on shade patches | 16:20 |
| *** lucas-hungry is now known as lucasagomes | 16:20 | |
| mordred | clarkb: oh - nevermind - wow, I see the real error | 16:20 |
| *** mhickey has quit IRC | 16:20 | |
| *** jamielennox is now known as jamielennox|away | 16:21 | |
| *** strigazi is now known as strigazi_AFK | 16:21 | |
| mordred | pabelanger, clarkb: http://logs.openstack.org/49/377049/1/check/gate-dsvm-nodepool-src-shade/e7e180b/logs/screen-nodepool-builder.txt.gz#_2016-09-26_21_12_56_700 | 16:21 |
| *** Guest46101 is now known as mgagne | 16:21 | |
| *** mgagne has quit IRC | 16:21 | |
| *** mgagne has joined #openstack-infra | 16:21 | |
| clarkb | ya it failed to connect for ubuntu image | 16:21 |
| jeblair | :( | 16:21 |
| clarkb | possibly related to therves thing maybe | 16:21 |
| mordred | possibly - it's also not using the pypi mirror in the dib build | 16:21 |
| *** tuannguyen has quit IRC | 16:22 | |
| *** caowei has quit IRC | 16:22 | |
| *** njohnston has quit IRC | 16:22 | |
| mordred | or maybe it would - it's failing downloading get-pip.py :) | 16:22 |
| *** ddieterly[away] is now known as ddieterly | 16:22 | |
| *** tphummel has joined #openstack-infra | 16:22 | |
| *** njohnston has joined #openstack-infra | 16:23 | |
| *** yamamoto has quit IRC | 16:23 | |
| *** tdasilva has quit IRC | 16:23 | |
| *** njohnston has quit IRC | 16:25 | |
| greghaynes | If you all can find a way to mirror that it should be easy to make dib used the mirrored URI for it... | 16:25 |
| *** flepied has quit IRC | 16:26 | |
| *** tuannguyen has joined #openstack-infra | 16:26 | |
| *** shardy has quit IRC | 16:26 | |
| *** jaosorior has quit IRC | 16:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on publishing updated rule to be on sync with website https://review.openstack.org/378769 | 16:28 |
| *** ihrachys has quit IRC | 16:29 | |
| *** yaume has quit IRC | 16:29 | |
| *** thorst_ is now known as thorst | 16:30 | |
| *** njohnston has joined #openstack-infra | 16:30 | |
| *** nzoueidi has quit IRC | 16:30 | |
| *** njohnston has quit IRC | 16:32 | |
| *** njohnston has joined #openstack-infra | 16:32 | |
| *** njohnston has quit IRC | 16:33 | |
| *** narayrak has quit IRC | 16:33 | |
| *** Swami_ has quit IRC | 16:34 | |
| *** njohnston has joined #openstack-infra | 16:34 | |
| *** jlanoux has quit IRC | 16:34 | |
| *** baoli has quit IRC | 16:34 | |
| *** sshnaidm|afk is now known as sshnaidm | 16:34 | |
| *** baoli has joined #openstack-infra | 16:35 | |
| *** njohnston has quit IRC | 16:35 | |
| *** armax has quit IRC | 16:36 | |
| *** annegentle has quit IRC | 16:36 | |
| *** njohnston has joined #openstack-infra | 16:36 | |
| *** Ahharu has joined #openstack-infra | 16:37 | |
| *** njohnston has quit IRC | 16:37 | |
| Ahharu | hi | 16:37 |
| Ahharu | 2016-09-28 16:25:10.462691 | cp: cannot stat Ă¢â‚¬Ëœ/root/openrcĂ¢â‚¬â„¢: No such file or directory | 16:37 |
| Ahharu | is this normal on beaker tests? | 16:37 |
| Ahharu | it just started bombing | 16:37 |
| *** tuannguyen has quit IRC | 16:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add caching decorators to zones and record_sets https://review.openstack.org/378069 | 16:38 |
| *** njohnston has joined #openstack-infra | 16:38 | |
| *** njohnston has quit IRC | 16:38 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on publishing updated rule to be on sync with website https://review.openstack.org/378769 | 16:39 |
| anteaya | Ahharu: well https://github.com/puppetlabs/beaker-rspec/pull/86 is apparently a cause of new issues | 16:39 |
| *** njohnston has joined #openstack-infra | 16:39 | |
| anteaya | Ahharu: what project are we discussing? | 16:39 |
| Ahharu | our project is `puppet-midonet` | 16:40 |
| Ahharu | https://review.openstack.org/#/q/puppet-midonet | 16:40 |
| *** Apoorva has joined #openstack-infra | 16:40 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on publishing updated rule to be on sync with website https://review.openstack.org/378769 | 16:40 |
| anteaya | https://review.openstack.org/#/c/378648/ merged two hours ago | 16:40 |
| *** njohnston has quit IRC | 16:40 | |
| *** esikachev has quit IRC | 16:40 | |
| *** njohnston has joined #openstack-infra | 16:40 | |
| anteaya | would that affect your project? | 16:40 |
| *** njohnston has quit IRC | 16:42 | |
| *** njohnston has joined #openstack-infra | 16:42 | |
| *** tuannguyen has joined #openstack-infra | 16:42 | |
| *** amitgandhinz has quit IRC | 16:43 | |
| *** njohnston has quit IRC | 16:43 | |
| *** njohnston has joined #openstack-infra | 16:45 | |
| *** drifterza has quit IRC | 16:45 | |
| *** njohnston has quit IRC | 16:45 | |
| anteaya | I'm going with you have enough to make you happy | 16:46 |
| *** njohnston has joined #openstack-infra | 16:47 | |
| *** njohnston has quit IRC | 16:47 | |
| cloudnull | pabelanger: clarkb: fungi: no errors in cloud1 for the last hour. seems like whatever was done has made things much happier. | 16:47 |
| *** slaweq_ has quit IRC | 16:48 | |
| anteaya | congratulations | 16:48 |
| *** njohnston has joined #openstack-infra | 16:50 | |
| *** krtaylor has joined #openstack-infra | 16:50 | |
| *** njohnston has quit IRC | 16:50 | |
| mordred | \o/ | 16:50 |
| *** njohnston has joined #openstack-infra | 16:51 | |
| anteaya | AJaeger: I see https://review.openstack.org/#/c/376573 merged, thank you | 16:51 |
| *** e0ne has quit IRC | 16:52 | |
| *** njohnston has quit IRC | 16:52 | |
| *** njohnston has joined #openstack-infra | 16:52 | |
| *** njohnston has quit IRC | 16:53 | |
| *** njohnston has joined #openstack-infra | 16:54 | |
| *** adrian_otto has quit IRC | 16:54 | |
| *** sandanar has joined #openstack-infra | 16:55 | |
| *** njohnston has quit IRC | 16:55 | |
| eharney | does the requirements update bot not automatically bump the hacking version in test-reqs.txt? i'd expect https://review.openstack.org/#/c/378793/ to contain the hacking<0.12 change | 16:55 |
| *** njohnston has joined #openstack-infra | 16:55 | |
| *** muawiakhan has joined #openstack-infra | 16:55 | |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate oscc cloud to Nodepool https://review.openstack.org/378561 | 16:56 |
| *** njohnston has quit IRC | 16:57 | |
| *** njohnston has joined #openstack-infra | 16:57 | |
| cloudnull | no rush, but any chance on getting https://review.openstack.org/#/c/378642, https://review.openstack.org/#/c/378215, https://review.openstack.org/#/c/378100/ though? | 16:57 |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate params and resources to Nodepool node https://review.openstack.org/378564 | 16:57 |
| *** amitgandhinz has joined #openstack-infra | 16:58 | |
| *** njohnston has quit IRC | 16:58 | |
| anteaya | eharney: any idea how long ago hacking changed? I'm trying to find the patch that changed it in requirements | 16:59 |
| *** njohnston has joined #openstack-infra | 16:59 | |
| *** muawiakh_ has joined #openstack-infra | 16:59 | |
| *** muawiakhan has quit IRC | 16:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr: Updated from global requirements https://review.openstack.org/378870 | 16:59 |
| clarkb | hacking is special in requirements | 16:59 |
| *** adrian_otto has joined #openstack-infra | 16:59 | |
| clarkb | because its expected to fail when bumped and require intervention | 16:59 |
| eharney | clarkb: ahh, i thought that might be the case. would be a good thing to note in the bot's commit message maybe, when it happens | 16:59 |
| anteaya | ah, thank you clarkb | 16:59 |
| clarkb | eharney: not sure what you mean the bot never makes that commit iirc | 17:00 |
| *** njohnston has quit IRC | 17:00 | |
| eharney | clarkb: i meant a message like "hacking's version has changed in test-reqs.txt, but it has been skipped" in the "Updated from global reqs" commit | 17:01 |
| *** njohnston has joined #openstack-infra | 17:01 | |
| clarkb | gotcha, feel free to bring it up with the requirements team and or push a patch | 17:01 |
| *** njohnston has quit IRC | 17:02 | |
| pabelanger | wow | 17:02 |
| clarkb | eharney: not sure if they have other plans around that | 17:02 |
| pabelanger | we just launched 1242 nodes in nodepool | 17:02 |
| *** eliqiao has joined #openstack-infra | 17:03 | |
| pabelanger | all from OpenStack proposal bot | 17:03 |
| *** njohnston has joined #openstack-infra | 17:04 | |
| clarkb | pabelanger: proposal bot launches nodes? | 17:04 |
| pabelanger | clarkb: sorry, the patchset from it did | 17:04 |
| pabelanger | https://review.openstack.org/#/q/topic:openstack/requirements | 17:05 |
| *** njohnston has quit IRC | 17:05 | |
| *** smarcet has joined #openstack-infra | 17:05 | |
| *** muawiakh_ has quit IRC | 17:06 | |
| pabelanger | cool, quota issue in infracloud-vanilla | 17:06 |
| *** njohnston has joined #openstack-infra | 17:06 | |
| pabelanger | OpenStackCloudException: Error in creating instance (Inner Exception: Quota exceeded for ram: Requested 8192, but already used 794624 of 800000 ram (HTTP 403) (Request-ID: req-9c729032-060f-4199-a455-80e1d77972df)) | 17:06 |
| *** muawiakhan has joined #openstack-infra | 17:06 | |
| fungi | yeah, when a requirements change lands, and the post job from that proposes requirements updates to other projects, each of those then spins up a number of check jobs | 17:06 |
| fungi | so tends to create a bit of a thundering herd | 17:06 |
| *** muawiakhan has quit IRC | 17:07 | |
| *** njohnston has quit IRC | 17:07 | |
| openstackgerrit | Merged openstack-infra/project-config: Make Horizon integration tests non-voting https://review.openstack.org/378406 | 17:07 |
| pabelanger | Ya | 17:07 |
| pabelanger | that's the first time we launched all nodes in infracloud-vanilla | 17:07 |
| *** psachin has quit IRC | 17:08 | |
| *** jpena is now known as jpena|off | 17:08 | |
| *** njohnston has joined #openstack-infra | 17:08 | |
| *** njohnston has quit IRC | 17:08 | |
| *** smarcet1981 has quit IRC | 17:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable networking-ovn plugin on subnode config https://review.openstack.org/377921 | 17:09 |
| anteaya | AJaeger: you are correct, I should have had robcresswell remove the non voting jobs from the gate in https://review.openstack.org/#/c/378406/ | 17:09 |
| *** njohnston has joined #openstack-infra | 17:09 | |
| anteaya | robcresswell: are you willing to offer a patch that cleans that up? | 17:10 |
| *** flepied has joined #openstack-infra | 17:12 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Replace *_ssh jobs with *_ipmitool https://review.openstack.org/377928 | 17:14 |
| *** ddieterly is now known as ddieterly[away] | 17:15 | |
| anteaya | I got my acronym wrong for cacti data retrevial, I so can't acronym | 17:16 |
| *** xarses has quit IRC | 17:19 | |
| pabelanger | rcarrillocruz: I | 17:20 |
| pabelanger | err | 17:20 |
| pabelanger | rcarrillocruz: I've bumped the ram in infracloud-vanilla (openstackzuul) to 1032192 | 17:21 |
| pabelanger | that lines up with 126 nodes at 8192 | 17:21 |
| pabelanger | rcarrillocruz: where did we land on setting quota in ansible? | 17:22 |
| openstackgerrit | Cory Benfield proposed openstack-infra/elastic-recheck: Prefer to invoke scripts directly without python. https://review.openstack.org/378925 | 17:22 |
| *** yamamoto has joined #openstack-infra | 17:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Enable container job on multinode https://review.openstack.org/288915 | 17:25 |
| *** berendt has quit IRC | 17:27 | |
| *** pvaneck has joined #openstack-infra | 17:27 | |
| *** sc68cal_ is now known as sc68cal | 17:27 | |
| *** Guest92 has quit IRC | 17:28 | |
| *** flepied has quit IRC | 17:29 | |
| *** sambetts is now known as sambetts|afk | 17:29 | |
| *** Swami has joined #openstack-infra | 17:31 | |
| *** yamamoto has quit IRC | 17:31 | |
| *** esberglu has quit IRC | 17:32 | |
| *** nadya has quit IRC | 17:32 | |
| *** esberglu has joined #openstack-infra | 17:32 | |
| *** tuannguyen has quit IRC | 17:33 | |
| *** apuimedo|away is now known as apuimedo | 17:34 | |
| *** xarses has joined #openstack-infra | 17:35 | |
| *** ralonsoh has quit IRC | 17:36 | |
| quade1 | AJaeger: Thanks for your help on this review. Do you know who I can reach out to to get Workflow+1? https://review.openstack.org/#/c/377908/ | 17:36 |
| *** nadya has joined #openstack-infra | 17:37 | |
| *** esberglu has quit IRC | 17:38 | |
| *** baoli has quit IRC | 17:39 | |
| *** baoli has joined #openstack-infra | 17:39 | |
| *** esberglu has joined #openstack-infra | 17:40 | |
| *** baoli has quit IRC | 17:40 | |
| *** baoli has joined #openstack-infra | 17:41 | |
| *** tuannguyen has joined #openstack-infra | 17:43 | |
| *** e0ne has joined #openstack-infra | 17:44 | |
| *** thorst has quit IRC | 17:45 | |
| *** thorst has joined #openstack-infra | 17:46 | |
| *** jimbaker` is now known as jimbaker | 17:48 | |
| *** quade1 has quit IRC | 17:48 | |
| *** amitgandhinz has quit IRC | 17:48 | |
| *** rbrndt has quit IRC | 17:48 | |
| *** quade has joined #openstack-infra | 17:49 | |
| *** amitgandhinz has joined #openstack-infra | 17:49 | |
| *** tpsilva has joined #openstack-infra | 17:50 | |
| *** fernnest has joined #openstack-infra | 17:50 | |
| *** fernnest_ has joined #openstack-infra | 17:50 | |
| *** lin_yang has quit IRC | 17:50 | |
| anteaya | fungi: 3 days ago someone uploaded a remove_device.php file to a forum thread for cacti that is 8 years old: http://forums.cacti.net/viewtopic.php?f=5&t=27713&start=15 | 17:53 |
| *** sshnaidm is now known as sshnaidm|afk | 17:53 | |
| anteaya | seems like removing a device from cacti isn't in the source code, which I find odd | 17:53 |
| *** adrian_otto has quit IRC | 17:54 | |
| fungi | normally they expect you to delete files off the filesystem | 17:54 |
| *** adrian_otto has joined #openstack-infra | 17:54 | |
| anteaya | is that how we removed stale host entries that have been removed from our system config? | 17:54 |
| *** adrian_otto has quit IRC | 17:55 | |
| fungi | i'm not sure, honestly. i haven't been doing it, so would need to explore | 17:56 |
| anteaya | okay yeah, that is what I got from your comment on friday | 17:56 |
| anteaya | was trying to be helpful | 17:56 |
| anteaya | if I'm wasting your time, I'll stop | 17:56 |
| AJaeger | quade: any project-config ;) core can help with 377909. Just ask here, those that review are on IRC. Somebody might read it... | 17:57 |
| *** trown is now known as trown|lunch | 17:57 | |
| *** amitgandhinz has quit IRC | 17:58 | |
| quade | AJaeger: Thanks again! Any core interested in a Workflow? https://review.openstack.org/#/c/377908/ | 17:58 |
| *** amitgandhinz has joined #openstack-infra | 17:59 | |
| *** annegentle has joined #openstack-infra | 17:59 | |
| *** adrian_otto has joined #openstack-infra | 17:59 | |
| AJaeger | pabelanger: could you look later at https://review.openstack.org/378062 , please? That'S a grafana change... | 17:59 |
| *** mriedem has quit IRC | 17:59 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 17:59 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 17:59 |
| clarkb | AJaeger: reviewing the manilla change now | 18:00 |
| *** lezbar has quit IRC | 18:01 | |
| clarkb | then cloudnull's changes | 18:01 |
| pabelanger | AJaeger: +2 | 18:01 |
| *** lezbar has joined #openstack-infra | 18:02 | |
| *** sandanar_ has joined #openstack-infra | 18:02 | |
| AJaeger | thanks, pabelanger and clarkb | 18:02 |
| *** amitgandhinz has quit IRC | 18:02 | |
| *** amitgandhinz has joined #openstack-infra | 18:03 | |
| *** rwsu has quit IRC | 18:03 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Make Horizon and D_O_A Django 1.10 tests voting https://review.openstack.org/378426 | 18:04 |
| *** esikachev has joined #openstack-infra | 18:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Add to my schedule optimization https://review.openstack.org/378944 | 18:04 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Add to my schedule optimization https://review.openstack.org/378944 | 18:05 |
| *** sandanar has quit IRC | 18:05 | |
| *** cardeois_ has joined #openstack-infra | 18:06 | |
| mtreinish | fungi, nibalizer, clarkb, mordred, pleia2, rcarrillocruz: if you get a sec could you take a look at: https://review.openstack.org/#/q/topic:mosquitto-redux which should clean a few things up | 18:06 |
| *** cardeoi__ has joined #openstack-infra | 18:06 | |
| mtreinish | I also think part of the reason we're seeing the websocket crash a bunch on mosquitto is because the only code example we have in the docs uses websockets | 18:07 |
| mtreinish | one of the patches in that topic adds an ssl code example (without websockets) which maybe will help with that | 18:07 |
| *** kzaitsev_mb has joined #openstack-infra | 18:08 | |
| *** cardeoi__ has quit IRC | 18:08 | |
| *** cardeois has quit IRC | 18:08 | |
| *** cardeois has joined #openstack-infra | 18:09 | |
| *** rbrndt has joined #openstack-infra | 18:10 | |
| *** cardeois_ has quit IRC | 18:10 | |
| clarkb | pabelanger: cloudnull I am good with merging the rebalancing of osic max-servers if you are both ready | 18:11 |
| clarkb | will review grafana changes first though | 18:12 |
| *** ddieterly[away] is now known as ddieterly | 18:12 | |
| pabelanger | clarkb: another 20mins and our last DIB for this day is finished, so if we could hold off on any nodepool restarts. But ready too | 18:12 |
| pabelanger | just creating qcow2 now | 18:12 |
| *** berendt has joined #openstack-infra | 18:13 | |
| clarkb | pabelanger: the rebalancing of max-servers won't require any restarts | 18:13 |
| clarkb | its just shifting load between the logical osic cloud1 providers | 18:14 |
| pabelanger | coolio | 18:14 |
| *** tqtran has joined #openstack-infra | 18:14 | |
| *** amoralej is now known as amoralej|off | 18:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Show more graphs in a single row https://review.openstack.org/378062 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use command module instead of zuul_runner https://review.openstack.org/369604 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Rename callback_plugins dir to callback https://review.openstack.org/371887 | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Rename zuul_runner to command https://review.openstack.org/369603 | 18:16 |
| *** ijw has quit IRC | 18:16 | |
| *** tdasilva has joined #openstack-infra | 18:16 | |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Add xenial tempest jobs https://review.openstack.org/377360 | 18:17 |
| *** jheroux has joined #openstack-infra | 18:18 | |
| *** tqtran has quit IRC | 18:18 | |
| *** Guest92 has joined #openstack-infra | 18:18 | |
| clarkb | cloudnull: one minor thing noted on https://review.openstack.org/#/c/378215/2 | 18:19 |
| *** Guest92_ has joined #openstack-infra | 18:20 | |
| clarkb | cloudnull: you ready for me to approve the rebalancing now or do you want to get the grafana changes in first? | 18:20 |
| irtermite | he's in a meeting clarkb | 18:20 |
| irtermite | one sec | 18:20 |
| *** degorenko is now known as _degorenko | 18:20 | |
| clarkb | irtermite: ok no rush | 18:20 |
| *** lezbar has quit IRC | 18:20 | |
| *** rossella_ has joined #openstack-infra | 18:20 | |
| *** lezbar has joined #openstack-infra | 18:21 | |
| irtermite | clarkb: go for rebalance; he doesn't need grafana quite yet | 18:23 |
| *** Guest92 has quit IRC | 18:23 | |
| sridhar_ram | infra-team: need a 2nd +2 for this change switching tacker python jobs voting https://review.openstack.org/#/c/370388/ ... anyone ? | 18:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Added additional (disk|s3500|s3700) to entries https://review.openstack.org/378215 | 18:24 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add support for extra elastic-search graph filter https://review.openstack.org/255585 | 18:25 |
| clarkb | irtermite: I went ahead and just fixed the grafana thing since it was a one character change | 18:25 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Make Elastic Recheck Watch more reusable https://review.openstack.org/251515 | 18:25 |
| *** Jeffrey4l has quit IRC | 18:28 | |
| *** baoli has quit IRC | 18:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Add osic-cloud1-(disk|s3500|3700) to grafana https://review.openstack.org/378642 | 18:28 |
| irtermite | thanks clarkb | 18:28 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Make uncategorized_fails.py configurable https://review.openstack.org/250072 | 18:28 |
| *** mriedem has joined #openstack-infra | 18:29 | |
| pabelanger | clarkb: okay, ready to support. nodepool uploading DIBs now | 18:29 |
| *** nadya has quit IRC | 18:30 | |
| mordred | kfox1111: was that it? | 18:31 |
| mrmartin | o/ hi | 18:31 |
| mordred | hey mrmartin | 18:32 |
| *** krtaylor has quit IRC | 18:32 | |
| mrmartin | hi mordred | 18:32 |
| *** nadya has joined #openstack-infra | 18:32 | |
| mrmartin | who knows what is the current situation of stackalytics.o.o ? | 18:32 |
| clarkb | I think fungi and pabelanger probably have the best udnersatnd of the current state of stackalytics | 18:33 |
| *** zz_dimtruck is now known as dimtruck | 18:33 | |
| *** ddieterly is now known as ddieterly[away] | 18:33 | |
| *** salv-orlando has quit IRC | 18:34 | |
| pabelanger | Ya, the work is done, its been running for a while. We just need to figure out the logistics on making the switch I think | 18:34 |
| pabelanger | I think we still need to do some tuning in apache2 too | 18:34 |
| mrmartin | do we have some tickets for the open issues in Storyboard? | 18:36 |
| pabelanger | nothing outside of https://storyboard.openstack.org/#!/story/2000274 | 18:37 |
| openstackgerrit | Merged openstack-infra/project-config: Update server distribution to HW specific entries https://review.openstack.org/378100 | 18:38 |
| clarkb | cloudnull: when you are out of meetings and have a moment I'd like to see if we can debug http://logs.openstack.org/19/373219/13/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/8f4250b/logs/devstack-subnodes-early.txt.gz basically ssh from one instance in osic cloud1 to another via the private ipv4 addresses seems to fail occasionally. We were seeing this on rackspace too | 18:38 |
| mrmartin | oh, ok I can allocate some time to finish this transition. | 18:38 |
| irtermite | ACK clarkb | 18:38 |
| mrmartin | so if we can collect what we need to do, I'm happy to help. | 18:38 |
| *** pcrews has quit IRC | 18:39 | |
| fungi | pabelanger: mrmartin: i think the main issue we have is that a ton of stackalytics analysis is driven from overrides in "configuration" which it can only update by restarting. but it doesn't maintain a persistent data backend and has to rebuild all of its analysis from scratch to populate the in-memory database it uses, so restarts take something like half an hour (complete outage) to complete | 18:40 |
| *** woodster_ has quit IRC | 18:40 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Enable more reasonable logging https://review.openstack.org/360164 | 18:40 |
| fungi | so we have a choice between frequent and lengthy outages for the service, or very infrequently applying updates to people's affiliation and address overrides | 18:41 |
| *** ijw has joined #openstack-infra | 18:41 | |
| fungi | or rewriting a big chunk of stackalytics to use a persistent db and/or handle live configuration updates without restarting | 18:41 |
| mrmartin | fungi: oh, ok, I'll try to deploy it in a vm, and check how it works actually, and how can we improve this persistent data backend. | 18:41 |
| mordred | fungi: I thought it put all of its stuff into memcached | 18:41 |
| openstackgerrit | Merged openstack/os-testr: Handle overlapping black regexes https://review.openstack.org/377999 | 18:41 |
| openstackgerrit | Merged openstack/os-testr: Updated from global requirements https://review.openstack.org/378870 | 18:42 |
| fungi | mordred: if so, then the cause of the lengthy outages to restart it may be for other reasons, but that was the story i had heard anyway | 18:42 |
| mordred | fungi: gotcha. I mean, I don't really actually know what I'm talking about :) | 18:42 |
| clarkb | fungi: anteaya the puppet change to remove debug logs on the jenkins log client is applied on logstash.o.o I will go ahead and restart the service to change its logging | 18:42 |
| clarkb | fungi: do you think I should also truncate the logs? | 18:43 |
| fungi | at least, i can tell you that if you restart stackalytics on stackalytics.o.o it's down for a long time, and the stackalytics.org maintainers have said they restart it manually on weekends when not as many people are looking at it | 18:43 |
| rcarrillocruz | pabelanger: https://github.com/ansible/ansible-modules-extras/pull/2709#issuecomment-248501356 | 18:43 |
| mordred | ahhh | 18:43 |
| rcarrillocruz | meaning, it's probably going to be for 2.3 | 18:43 |
| mrmartin | yeah, ok | 18:43 |
| rcarrillocruz | so i tell you what | 18:43 |
| *** yamahata has joined #openstack-infra | 18:43 | |
| mrmartin | cron can do magic | 18:43 |
| rcarrillocruz | i'll just hack around in the launcher parsing quota resources | 18:43 |
| pabelanger | fungi: we try to dump the data from memory if possible, but ya. It is an issue | 18:43 |
| rcarrillocruz | and run 'osc' commands in the background | 18:43 |
| rcarrillocruz | i'll let you know when i have something | 18:43 |
| pabelanger | rcarrillocruz: ah, okay | 18:44 |
| rcarrillocruz | fungi: in regards to the cacti replacement, should I put it on RAX_IAD or other region? | 18:44 |
| fungi | pabelanger: oh, is it doing a pickle to disk or something? | 18:44 |
| fungi | rcarrillocruz: rax dfw, just like all our other control plane servers | 18:44 |
| rcarrillocruz | ah yeah | 18:44 |
| rcarrillocruz | i keep on confusing regions | 18:44 |
| rcarrillocruz | thx | 18:44 |
| fungi | me too, no worries | 18:44 |
| pabelanger | fungi: no, I wrote a crontab / logrotate to handle it: http://git.openstack.org/cgit/openstack-infra/puppet-stackalytics/tree/manifests/init.pp#n127 | 18:45 |
| fungi | clarkb: we likely don't care about the logs from the past 24 hours, so no need to truncate them, just delete them? | 18:45 |
| *** vsaienko has joined #openstack-infra | 18:45 | |
| rcarrillocruz | the way i did replacements in gozer was: create a cacti2 machine, add the puppet node, run puppet, do migration from cacti to cacti2, decommission cacti, point DNS of cacti.openstack.org to cacti2 machine, rename cacti2 to cacti on openstack . Is that how you usually do replacements? | 18:45 |
| vsaienko | devstack-gate core: please help to review/merge https://review.openstack.org/#/c/335981 | 18:45 |
| clarkb | fungi: ya I deleted them and service is restarted | 18:46 |
| *** sdake has quit IRC | 18:46 | |
| *** ijw has quit IRC | 18:46 | |
| fungi | rcarrillocruz: we just create a second one named cacti.openstack.org in nova and keep track of ip addresses until we update dns | 18:46 |
| mordred | I may not have realized you can rename servers in openstack though | 18:47 |
| rcarrillocruz | eugh, and doesn't the ansible wheel do weird things with two servers being named the same ? | 18:47 |
| fungi | rcarrillocruz: nope. ansible is working via uuid | 18:47 |
| mordred | or, it will for anything with duplicate names | 18:47 |
| fungi | rcarrillocruz: unless you want to do the additional work of making puppet handle maintaining/grouping cacti01 as well, and then we can leave the replacement named cacti01.openstack.org and create a cacti.openstack.org cname to it once it's running | 18:47 |
| rcarrillocruz | mordred: yeah, i even have some PR for an os_server_actions rename thing on my laptop | 18:47 |
| rcarrillocruz | i have to test it more thoroughly and push it, i hacked in it precisely having upgrades in mind | 18:48 |
| *** krtaylor has joined #openstack-infra | 18:48 | |
| fungi | rcarrillocruz: the problem with "renaming" servers after creation is that there are lots of places the original hostname ends up embedded on the filesystem | 18:48 |
| rcarrillocruz | yeah, an ansible playbook to reconcile new name would be good for that use case | 18:49 |
| mordred | yah | 18:49 |
| rcarrillocruz | /etc/hostname | 18:49 |
| rcarrillocruz | /etc/hosts | 18:49 |
| *** rossella_ has quit IRC | 18:49 | |
| rcarrillocruz | 18:49 | |
| rcarrillocruz | whatever | 18:49 |
| fungi | ssh host keys too | 18:49 |
| fungi | and possibly lots of other stuff depending on what's initially installed | 18:50 |
| rcarrillocruz | so i guess that's part of the things to fix to end up with our goal of having uniquely named servers, yah? | 18:50 |
| fungi | in a lot of cases it's just embedded comments or other benign strings that don't influence operation | 18:50 |
| pabelanger | I thought doing cacti01.o.o and cacti02.o.o help solve some of those problems | 18:50 |
| clarkb | vsaienko: quick qestuon about that, set ips is set to false which means you won't have any l3 interface for that bridge on the test nodes. Which should mean you can't actually route to those IPs | 18:50 |
| clarkb | vsaienko: does that need to be set to True? | 18:50 |
| *** rhallisey has quit IRC | 18:50 | |
| fungi | rcarrillocruz: if you go the route with an ordinal suffix on the hostname (which i've done for a couple of our replacements now since we discussed it) you need to make sure to transition hiera from fqdn to a group, as well as making sure we get rid of any fqdn-isms in the puppet modules/classes we're using | 18:51 |
| *** rhallisey has joined #openstack-infra | 18:51 | |
| clarkb | vsaienko: or is l2 sufficient for you? | 18:51 |
| vsaienko | clarkb: we tested this configuration it works just fine, we need only L2 https://review.openstack.org/#/c/368173/ | 18:52 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Enable more reasonable logging https://review.openstack.org/360164 | 18:52 |
| clarkb | vsaienko: I only see failing jobs because devstack exercises could not be found? | 18:53 |
| clarkb | oh thats multitest not multinode | 18:53 |
| rcarrillocruz | fungi: yeah, would be good to hack on that, cos i see there are streams of ansibling that machinery | 18:53 |
| rcarrillocruz | which can potentially be reused for future replacements | 18:53 |
| vsaienko | clarkb: please have look at experimental multinode job: gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ssh-tinyipa-multinode-nvSUCCESS in 49m 53s (non-voting) | 18:54 |
| *** annegentle has quit IRC | 18:54 | |
| clarkb | vsaienko: ya I am looking at it now | 18:54 |
| clarkb | vsaienko: this uses the ssh agent which ssh's to the host with the fake baremetal then boots using libvirt. Then from there how is ironic talking to it? | 18:55 |
| vsaienko | clarkb: https://review.openstack.org/#/c/368647/3/devstack/lib/ironic@1551 | 18:56 |
| *** lezbar has quit IRC | 18:56 | |
| *** baoli has joined #openstack-infra | 18:56 | |
| vsaienko | clarkb: we plug fake bare metal VMs to this network https://review.openstack.org/#/c/368647/3/devstack/lib/ironic@1545 | 18:57 |
| *** lezbar has joined #openstack-infra | 18:57 | |
| clarkb | vsaienko: right but what sort of communication is happening there? I am trying to figure out if that is required at all or if you can just use the neutron network. Because I am surprised if l2 without any ip addressing is suffiicent for you | 18:58 |
| clarkb | since things like pxe require ip addresses iirc | 18:58 |
| vsaienko | clarkb: I did near 10 recheck to make sure that conductor on primary node is able to talk to VMs located on subside | 18:58 |
| clarkb | vsaienko: yes I believe that it is working. I am wondering how | 18:59 |
| vsaienko | clarkb: from the tunnel we need only L2, regarding to communication VM receives IP from neutron and than talks with conductor and API via L2 | 18:59 |
| vsaienko | clarkb: I'm sorry via L3 | 18:59 |
| clarkb | vsaienko: what traffic goes over the ironic tunnel? | 18:59 |
| vsaienko | dhcp, pxe boot, http | 19:00 |
| clarkb | vsaienko: if neutron manages the l3 then why do we need a separate l2 network is I guess what this boils down to | 19:00 |
| clarkb | right thats not just l2 | 19:00 |
| clarkb | pxe and http require ip addresses | 19:00 |
| clarkb | I guess maybe the test nodes are arping on all interfaces? | 19:00 |
| vsaienko | clarkb: I tried with native neutron tunnel, but it wasn't working, I think it is due to neutron ovs security flows | 19:00 |
| *** hogepodge has quit IRC | 19:01 | |
| clarkb | i would expect it to arp out the default route though if the ip addrs for the VMs aren't in a range it otherwise knows about which shouldn't get any ersponse | 19:01 |
| vsaienko | clarkb: right, VM receives L3 info from Neutron, and that it communicates with ironic services via L3 | 19:01 |
| *** spzala has quit IRC | 19:01 | |
| clarkb | vsaienko: yes that part I understand. The part that is confusing me is how this l2 only bridge manages to do anything useful | 19:02 |
| *** trown|lunch is now known as trown | 19:03 | |
| *** tuannguyen has quit IRC | 19:03 | |
| vsaienko | clarkb: it allows to pass L3 traffic between VM and host IP on another node (ironic api/conductor listening ) on that IP, but Neutron doesn't know about it | 19:04 |
| clarkb | vsaienko: but the nodes don't know about it either bceause you are not assigning an address to those bridges on the nodes | 19:04 |
| clarkb | vsaienko: the only things that will know about it are VMs on the bridge | 19:04 |
| *** inc0 has joined #openstack-infra | 19:04 | |
| clarkb | does ironic create a port itself for that? | 19:05 |
| *** baoli has quit IRC | 19:05 | |
| clarkb | s/port/interface/ | 19:05 |
| *** baoli has joined #openstack-infra | 19:06 | |
| *** tqtran has joined #openstack-infra | 19:06 | |
| vsaienko | clarkb: yes, I think this code you are asking for https://review.openstack.org/#/c/368644/3/devstack/lib/ironic@579 | 19:06 |
| pabelanger | clarkb: mordred: this is from osic-cloud1 right now: http://paste.openstack.org/show/583365/ | 19:06 |
| *** dtardivel has quit IRC | 19:07 | |
| pabelanger | guess we need to make nodepool check if image is currently queue / saving, and skip upload? | 19:07 |
| *** berendt has quit IRC | 19:08 | |
| clarkb | vsaienko: yes thats it. So you do have to assign an address you have just chosen to not do it in devstack-gate | 19:08 |
| *** kzaitsev_mb has quit IRC | 19:08 | |
| rcarrillocruz | pabelanger: in regards to launcher things, can you review https://review.openstack.org/#/c/378552/ ? for adding roles support, now with tests | 19:08 |
| rcarrillocruz | mordred: ^ | 19:08 |
| clarkb | vsaienko: maybe you can leave a followup change to add a comment that describes that? | 19:08 |
| kfox1111 | is there a way to check if a job is running or stuck or waiting to run? | 19:08 |
| irtermite | If you all see issues that need tracked or cloudnull or I are not around... feel free to drop an issue https://github.com/osic/osic-clouds | 19:08 |
| vsaienko | clarkb: yes, I we wan't do it in devstack-gate because we already doing it in ironic | 19:08 |
| kfox1111 | 378056,19 has a 2hr 10min on it. and should onlyl take a few minutes to run. | 19:08 |
| clarkb | vsaienko: ya I think its fine, its just confusing fron devstack-gate's perspective | 19:09 |
| vsaienko | clarkb: we don't want to assign in devstack-gatse | 19:09 |
| kfox1111 | though I gues everythign else looks like its got similar runtimes on it... | 19:09 |
| pabelanger | rcarrillocruz: +2, the test helped | 19:09 |
| clarkb | kfox1111: click on the blue box and it will expand to give you job lists | 19:09 |
| rcarrillocruz | thx | 19:09 |
| kfox1111 | mordred: not sure... jobs been stuck for a while. :/ | 19:09 |
| quade | Any cores willing to Workflow this bad boy? =D https://review.openstack.org/#/c/377908/ | 19:10 |
| rcarrillocruz | i'll add tests for the user_roles change | 19:10 |
| kfox1111 | clarkb: oh wow. didnt know yo ucould do that. :) | 19:10 |
| kfox1111 | thanks. :) | 19:10 |
| *** mhickey has joined #openstack-infra | 19:11 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: No need to make sure ZK lock path exists https://review.openstack.org/378961 | 19:11 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 19:12 |
| *** berendt has joined #openstack-infra | 19:12 | |
| *** yamahata has quit IRC | 19:13 | |
| vsaienko | clarkb: many thanks for your help, could you please help also to review/merge patches in the chain: https://review.openstack.org/#/c/352790 once you have a free time of course | 19:13 |
| clarkb | vsaienko: jlvillal has comments on that particular change | 19:14 |
| fungi | infra-root: any objections to https://review.openstack.org/378686 ? (adds ssh access for jpmaxman to openstackid.org so he can double-up with smarcet on locally troubleshooting some elusive errors they're encountering under load) | 19:14 |
| *** ddieterly[away] is now known as ddieterly | 19:14 | |
| *** cardeois has quit IRC | 19:15 | |
| vsaienko | clarkb: I will reupload last patch in the chain, there are two more patches https://review.openstack.org/#/c/364830 https://review.openstack.org/#/c/368611 | 19:15 |
| *** cardeois has joined #openstack-infra | 19:15 | |
| rcarrillocruz | fungi: no objections, the more help the better | 19:15 |
| clarkb | vsaienko: ya reviewing them really quickly so you can update all in one go | 19:16 |
| *** gyee has quit IRC | 19:16 | |
| mordred | fungi: fine by me - I +2'd but did not +A so that you can gather as much feedback as you like | 19:16 |
| *** pcrews has joined #openstack-infra | 19:17 | |
| vsaienko | clarkb: thank you, I will ping you when it done | 19:17 |
| rcarrillocruz | so folks, are we good to have chocolate on nodepool so we can get images building going ? | 19:17 |
| rcarrillocruz | https://review.openstack.org/#/c/378559/1 | 19:17 |
| clarkb | vsaienko: for the one that enables services you want the api to be running on the subnode too? I don't think it will be used there | 19:17 |
| clarkb | vsaienko: since we don't have a load balancer configured only the api services running on the primary node should show up in the catalog I think? maybe nova round robins them when talking to ironic though? | 19:18 |
| vsaienko | clarkb: yeah, I'm going to remove api from subnode also | 19:18 |
| *** shu-mutou-AWAY has quit IRC | 19:20 | |
| rcarrillocruz | clarkb , pabelanger , fungi ^ | 19:20 |
| clarkb | vsaienko: looks like you have overloaded what multihost means too? | 19:21 |
| clarkb | vsaienko: multihost should specifically mean nova's multihost setting which is not useable with neutron and ironic aiui | 19:21 |
| pabelanger | rcarrillocruz: sure, but I'd like to hold off on restarting nodepool-builder until later tonight | 19:22 |
| *** salv-orlando has joined #openstack-infra | 19:22 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data https://review.openstack.org/378969 | 19:22 |
| rcarrillocruz | k, since you'll be around after i go to bed, i leave that to you approving then? | 19:22 |
| *** sandanar_ has quit IRC | 19:22 | |
| pabelanger | rcarrillocruz: Ya, +3 now, I'll restart the builder later tonight | 19:23 |
| rcarrillocruz | ++, thanks | 19:23 |
| *** vsaienko has quit IRC | 19:24 | |
| *** vsaienko has joined #openstack-infra | 19:25 | |
| mtreinish | clarkb: https://newsroom.t-mobile.com/news-and-blogs/t-mobiles-endless-summer.htm 2 weeks too late. That would have been nice to have in germany | 19:25 |
| alexm2 | AJaeger: could you add workflow to https://review.openstack.org/#/c/377908/ please? | 19:25 |
| AJaeger | alexm2: no. | 19:26 |
| *** tqtran has quit IRC | 19:26 | |
| AJaeger | alexm2: two cores need to +2. | 19:26 |
| *** kzaitsev_mb has joined #openstack-infra | 19:26 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data https://review.openstack.org/378969 | 19:26 |
| clarkb | mtreinish: will come in handy in barcelona though | 19:26 |
| *** eeiden has quit IRC | 19:26 | |
| *** skipp has quit IRC | 19:26 | |
| *** stew925 has quit IRC | 19:26 | |
| *** wcriswell has quit IRC | 19:26 | |
| alexm2 | AJaeger, got it. Who can I ask? :) | 19:26 |
| *** vsaienko has quit IRC | 19:28 | |
| *** Guest92_ has quit IRC | 19:28 | |
| *** vsaienko has joined #openstack-infra | 19:29 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data https://review.openstack.org/378969 | 19:30 |
| *** tqtran has joined #openstack-infra | 19:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Return build or upload number with recent data https://review.openstack.org/378969 | 19:32 |
| *** Guest92 has joined #openstack-infra | 19:32 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Create Member role on infracloud clouds https://review.openstack.org/378976 | 19:33 |
| *** sdake has joined #openstack-infra | 19:34 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Mirror centos virt mirror to AFS https://review.openstack.org/378978 | 19:35 |
| EmilienM | dmsimard: ^ | 19:35 |
| * clarkb wanders off to find lunch | 19:35 | |
| *** lezbar has quit IRC | 19:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add chocolate provider to Nodepool https://review.openstack.org/378559 | 19:36 |
| *** lezbar has joined #openstack-infra | 19:36 | |
| fungi | alexm2: i'm curious, what is JIRA:NCP-2067 supposed to mean in the commit message of https://review.openstack.org/377908 ? | 19:39 |
| alexm2 | fungi: it's our internal ticketing system id | 19:40 |
| fungi | it seems a bit strange to make vague references to what i can only assume is an unreachable, proprietary bug tracker in an upstream repo of a free software project | 19:40 |
| fungi | alexm2: do you mind if i edit the commit message to remove that before i approve the change? | 19:41 |
| alexm2 | fungi: go ahead | 19:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Make docs non-voting for quark, remove template https://review.openstack.org/377908 | 19:42 |
| *** eeiden has joined #openstack-infra | 19:43 | |
| *** salv-orl_ has joined #openstack-infra | 19:43 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: tripleo.sh: log overcloud deploy command arguments before deploy https://review.openstack.org/370632 | 19:43 |
| alexm2 | fungi: thanks! | 19:44 |
| fungi | you're welcome | 19:44 |
| *** nherciu has quit IRC | 19:44 | |
| *** oanson has joined #openstack-infra | 19:45 | |
| *** skipp has joined #openstack-infra | 19:45 | |
| *** stew925 has joined #openstack-infra | 19:46 | |
| *** camunoz_ has joined #openstack-infra | 19:46 | |
| *** salv-orlando has quit IRC | 19:46 | |
| *** wcriswell has joined #openstack-infra | 19:46 | |
| sdague | um, what's up with the graphite graphs for nodepool? | 19:48 |
| pabelanger | mordred: clarkb: jeblair: now that we have runtime-ssh-host-keys in diskimage-builder, I'd like to land: https://review.openstack.org/#/c/374371/ which removes that functionality from glean | 19:48 |
| sdague | it's like 1000 nodes available but jobs waiting an hour to get a test node? | 19:48 |
| pabelanger | sdague: we suspect nodepool cannot keep up with the demand now. jeblair has some patches to break nodepool into multiple daemons now | 19:49 |
| *** dimtruck is now known as zz_dimtruck | 19:49 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add imageUploadLock() API https://review.openstack.org/378983 | 19:49 |
| *** vsaienko has quit IRC | 19:50 | |
| *** vsaienko has joined #openstack-infra | 19:51 | |
| *** camunoz_ has quit IRC | 19:51 | |
| *** rwsu has joined #openstack-infra | 19:53 | |
| fungi | jeblair: pabelanger: all the split-daemon patches so far are merged at this point, right? i'm not finding any more of them open for review at least | 19:54 |
| pabelanger | fungi: clarkb: apparently ssh-keygen -A requires some keyboard inputs: http://paste.openstack.org/show/583376/ | 19:56 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic https://review.openstack.org/364830 | 19:56 |
| pabelanger | fungi: clarkb: a quick googles suggests doing $ yes n | ssh-keygen -A | 19:56 |
| pabelanger | to skip overriding of keys | 19:56 |
| pabelanger | overwriting* | 19:56 |
| pabelanger | fungi: I think all patches are merged | 19:57 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update ENABLED_SERVICE on subnode with ironic https://review.openstack.org/368611 | 19:57 |
| pabelanger | fungi: clarkb: I never seen the prompt in my testing, because I was also running: https://review.openstack.org/#/c/374371/ | 19:58 |
| *** vsaienko has quit IRC | 19:59 | |
| fungi | pabelanger: oh! got it. in your testing there weren't existing host keys because dib was deleting them and glean was not readding them before your initscript ran ssh-keygen | 20:01 |
| *** spzala has joined #openstack-infra | 20:01 | |
| greghaynes | It'd be really neat to have some kind of functional testing for this | 20:02 |
| greghaynes | I dont think it'd be too hard to do either | 20:02 |
| fungi | pabelanger: but without 374371 in production glean is (at least sometimes) creating host keys before the initscript runs ssh-keygen, and then you get prompted to overwrite them | 20:02 |
| pabelanger | fungi: correct, I'm testing yes n | ssh-keygen -A now, but think that is the way to disable the prompt | 20:03 |
| *** kzaitsev_mb has quit IRC | 20:03 | |
| fungi | pabelanger: and i agree, there doesn't seem to be an option to ssh-keygen to force overwriting :/ | 20:03 |
| greghaynes | I dont think you want force overwriting? | 20:03 |
| *** berendt has quit IRC | 20:04 | |
| fungi | well, optionally we can make that initscript only create host keys when they don't already exist, correct | 20:04 |
| *** lucasagomes is now known as lucas-afk | 20:04 | |
| greghaynes | right, if you force overwrite I think we break users who want to reboot and not change hostkeys | 20:04 |
| fungi | so the initscript can just no-op and succeed if the keys are already there, and that satisfies the dependency the sshd initscript has | 20:04 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Handle ssh-keygen prompts for existing keys https://review.openstack.org/378985 | 20:05 |
| pabelanger | need to build an image to test that | 20:05 |
| fungi | greghaynes: i agree. in our case (where the machine isn't reused and we don't support subsequent rebooting) it's mostly irrelevant, but in a general sense the initscript should no-op when the keys already exist | 20:05 |
| greghaynes | actually, wait a sec - http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/runtime-ssh-host-keys/init-scripts/systemd/ssh-keygen.service | 20:05 |
| greghaynes | Those ConditionPathExists should prevent this from happening | 20:06 |
| pabelanger | greghaynes: right, so far I am only seeing this in ubuntu-trusty | 20:06 |
| pabelanger | which is upstart | 20:06 |
| greghaynes | ah | 20:06 |
| *** spzala has quit IRC | 20:07 | |
| *** berendt has joined #openstack-infra | 20:07 | |
| *** salv-orl_ has quit IRC | 20:07 | |
| *** salv-orlando has joined #openstack-infra | 20:08 | |
| fungi | no way to make upstart do the same? | 20:08 |
| pabelanger | sadly no | 20:08 |
| *** ddieterly is now known as ddieterly[away] | 20:09 | |
| pabelanger | we'd have to add our logic as a pre-script I think | 20:09 |
| *** inc0 has quit IRC | 20:09 | |
| *** flepied has joined #openstack-infra | 20:09 | |
| pabelanger | build started | 20:10 |
| openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/378788 | 20:11 |
| pabelanger | clarkb: I think the scenario you mentioned this morning about deleting an DIB just happened | 20:15 |
| pabelanger | that is why we are seeing a lot of failures in osic-cloud1-s3500 | 20:15 |
| pabelanger | clarkb: actually, the image is still there... Hmm | 20:16 |
| *** ddieterly[away] is now known as ddieterly | 20:17 | |
| pabelanger | okay, osic-cloud1 uploaded a newer one | 20:18 |
| pabelanger | I've deleted it, since that region is going offline | 20:18 |
| *** oanson has quit IRC | 20:20 | |
| *** roxanaghe has quit IRC | 20:22 | |
| *** tonytan4ever has quit IRC | 20:23 | |
| *** mhickey has quit IRC | 20:23 | |
| *** annegentle has joined #openstack-infra | 20:25 | |
| *** Goneri has quit IRC | 20:27 | |
| *** baoli has quit IRC | 20:29 | |
| *** trukise has joined #openstack-infra | 20:29 | |
| pabelanger | fungi: yes n | ssh-keygen -A didn't work as expected | 20:30 |
| pabelanger | fungi: I guess we could not use ssh-keygen -A and manually generate any files that don't exists | 20:31 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Prefer to invoke scripts directly without python. https://review.openstack.org/378925 | 20:31 |
| fungi | pabelanger: have you ruled out being able to test for file existence in the upstart config? do we just need it to wrap a "very small shell script"? | 20:32 |
| mordred | pabelanger, greghaynes, fungi, clarkb: btw - I spoke with Jan van Eldik from CERN at OpenStack Nordic ... he's potentially interested in using glean there | 20:32 |
| greghaynes | ooo | 20:33 |
| pabelanger | fungi: that is what I am testing now, | 20:33 |
| fungi | pabelanger: oh, i see above you already said that, something about it having to be a pre-script | 20:33 |
| *** Apoorva_ has joined #openstack-infra | 20:33 | |
| mordred | in chatting with him, the split between the things in the simple-init element and the things in the glean repo struck me and made me wonder if we're making it harder for people to use it - but I betcha we can just get feedback from Jan on that | 20:33 |
| fungi | mordred: woah! for science!!! | 20:33 |
| mordred | yah | 20:33 |
| greghaynes | mordred: yea, I can definitely see that | 20:34 |
| *** ddieterly is now known as ddieterly[away] | 20:34 | |
| *** ddieterly[away] is now known as ddieterly | 20:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:34 | |
| greghaynes | IMO dib should just be some installer glue for the thing so the more we can make glean responsible for there the better | 20:34 |
| greghaynes | random thought - maybe the new ssh-key element can grab the init scripts from glean | 20:35 |
| *** Apoorva has quit IRC | 20:36 | |
| mordred | hrm - maybe so | 20:36 |
| fungi | well, part of teh challenge here is that dib is clearing out those keys, sshd on most platforms won't start without them present, so we need the sshd initscript-like-things to be patched to depend on the key creation initscript-like-things | 20:37 |
| *** adrian_otto has quit IRC | 20:37 | |
| clarkb | sshd will start it will just derp connections and not be useful | 20:38 |
| fungi | so if dib is installing openssh-server or whatever, it should probably be responsible for patching its initscripts, which only makes sense if it also ships the initscripts it's patching them to depend on | 20:38 |
| greghaynes | hrmmm | 20:38 |
| mordred | AFS isn't the solutoin to this is it? | 20:39 |
| clarkb | I thought we didnt need to patch sshds scripts | 20:39 |
| clarkb | we just add init things that ho before sshd | 20:39 |
| fungi | mordred: only if coupled with nntp | 20:39 |
| clarkb | *go before | 20:39 |
| fungi | clarkb: can you do that with lsb initscript headers? or are we really only needing to be able to declare it with systemd and upstart, not sysvinit? | 20:40 |
| clarkb | not sure about sysv but none of the distros we support sysv so not sure if it matters | 20:40 |
| fungi | okay, so at least with upstart and systemd we should be able to declare a "before" relationship in the keygen init config | 20:41 |
| clarkb | yes | 20:41 |
| fungi | in which case ignore what i said about patching sshd initscripts | 20:41 |
| fungi | oldthink | 20:41 |
| pabelanger | fungi: we can likely just do something like: http://paste.openstack.org/show/583381/ for upstart, rather depending on -A | 20:41 |
| fungi | i suppose the glean package could just ship them in tat case, yes | 20:42 |
| *** berendt has quit IRC | 20:42 | |
| jeblair | pabelanger: what's the status of the image uploads you were waiting on? | 20:42 |
| pabelanger | jeblair: finished | 20:43 |
| fungi | pabelanger: that's probably fine, yes | 20:43 |
| jeblair | so, okay to restart nodepool soon? | 20:43 |
| *** ddieterly is now known as ddieterly[away] | 20:43 | |
| pabelanger | jeblair: okay on this side | 20:43 |
| fungi | i'm fine with a nodepoold restart any time | 20:43 |
| jeblair | ok, i'll do that in around 20 mins | 20:43 |
| *** rfolco_ has quit IRC | 20:44 | |
| *** tpsilva has quit IRC | 20:44 | |
| *** jamielennox|away is now known as jamielennox | 20:45 | |
| *** jkilpatr has quit IRC | 20:45 | |
| cloudnull | irtermite: clarkb: sorry been in meetings for a good part of today. I'm looking into the "Failed to connect to the host via ssh" issue now. | 20:46 |
| mordred | cloudnull: I'd like to suggest that meetings are not a great use of your time :) | 20:47 |
| mordred | cloudnull: if it would be helpful for me to express that to other people, I'd be happy to | 20:47 |
| cloudnull | ++ I completely agree. | 20:47 |
| * mordred is good at loudly expressing things at people | 20:47 | |
| cloudnull | i have a flag on my desk that says "I survived another meeting that should've been an email". | 20:47 |
| cloudnull | sadly it does not get the point across. | 20:48 |
| mordred | cloudnull: maybe you should schedule a meeting to discuss that point | 20:48 |
| *** tuannguyen has joined #openstack-infra | 20:49 | |
| *** spzala has joined #openstack-infra | 20:49 | |
| cloudnull | that is a good idea. | 20:49 |
| *** burgerk has quit IRC | 20:49 | |
| *** esikachev has quit IRC | 20:50 | |
| fungi | back when i was forced to maintain a schedule in a group calendaring system, i would preemptively book most of my days and leave a few slots open where people could invite me to meetings | 20:50 |
| cloudnull | we might need to schedule a meeting to talk about the meeting to discuss the points about why I dont need to be in meetings all day | 20:50 |
| fungi | the majority of my schedule was "i'm getting work done, don't interrupt me" | 20:51 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add cacti2 replacement node to Puppet https://review.openstack.org/378996 | 20:52 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for upstart https://review.openstack.org/378985 | 20:52 |
| *** spzala has quit IRC | 20:53 | |
| *** nadya has quit IRC | 20:53 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout/puppet: run scenario003 for puppet-designate https://review.openstack.org/378997 | 20:54 |
| EmilienM | dmsimard: ^ | 20:54 |
| johnsom | Hi infra folks. Diskimage-builder is grabbing pip from https://bootstrap.pypa.io/get-pip.py in this element: https://github.com/openstack/diskimage-builder/blob/master/elements/pip-and-virtualenv/source-repository-pip-and-virtualenv | 20:54 |
| clarkb | cloudnull: cool, I can get uuids for those instances if you think it will help | 20:54 |
| dmsimard | EmilienM: cool | 20:54 |
| johnsom | Do we have a cache URL we should use for this in gate jobs? | 20:54 |
| clarkb | johnsom: we do I would haev to look at our nodepool element that caches things for devsatck to find the cache path though | 20:54 |
| dmsimard | EmilienM: added a comment | 20:55 |
| johnsom | Ok, yeah, we noticed failures getting this url in some of our gate jobs. | 20:55 |
| greghaynes | get-pip must have had some big outage, image building failed due to that too | 20:55 |
| EmilienM | dmsimard: we can't | 20:56 |
| clarkb | jeblair: I am back to a computer post lunch too and can help with nodepool if necessary | 20:56 |
| EmilienM | dmsimard: only zuul v3 will have this feature | 20:56 |
| dmsimard | EmilienM: so reviews to puppet-designate on mitaka would fail ? | 20:56 |
| dmsimard | EmilienM: or no, it just won't have designate in them I guess | 20:56 |
| EmilienM | nope, they'll just run the scenario003 :) | 20:56 |
| dmsimard | ok, sure, whatever | 20:56 |
| EmilienM | yes, without designate | 20:56 |
| greghaynes | clarkb: johnsom /home/jenkins/cache/files/get-pip.py | 20:56 |
| EmilienM | we'll review our layout in Zuul v3 | 20:57 |
| *** tqtran_ has joined #openstack-infra | 20:57 | |
| greghaynes | so DIB_REPOLOCATION_pip_and_virtualenv /home/jenkins/cache/files/get-pip.py should do the trick | 20:58 |
| greghaynes | er, = not <space> | 20:58 |
| johnsom | greghaynes Cool, I will give that a try | 20:58 |
| johnsom | Yep | 20:58 |
| *** raildo has quit IRC | 20:58 | |
| *** tqtran has quit IRC | 20:59 | |
| cloudnull | clarkb: if you have a few UUIDs handy it'd be great to work from those hosts and try and recreate some of those issues. | 20:59 |
| *** tuannguyen has quit IRC | 20:59 | |
| mordred | "Your team doesn't need to worry about DNS, editing records, CNAMEs or ALIAS. Just focus on shipping." | 20:59 |
| mordred | I feel like the industry is spending so much trying to make sure that engineers don't ever have to worry about engineering | 20:59 |
| *** tuannguyen has joined #openstack-infra | 20:59 | |
| *** ijw has joined #openstack-infra | 21:00 | |
| clarkb | cloudnull: http://logs.openstack.org/19/373219/13/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/f0f5d66/console.html#_2016-09-28_20_40_00_282993 looks like we saw similar there too | 21:00 |
| clarkb | cloudnull: ya let me dig up a couple paris of uuids for you | 21:00 |
| *** Marx314 has joined #openstack-infra | 21:00 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Enable container job on multinode https://review.openstack.org/288915 | 21:01 |
| cloudnull | that'd be great ! thanks | 21:01 |
| *** ijw has quit IRC | 21:01 | |
| *** trown is now known as trown|outtypewww | 21:01 | |
| clarkb | cloudnull: 3cb4a28b-9c3c-40d8-9832-72f6757f5b6a and 640ae349-970d-4ac5-9220-c4b72cebd4f7 | 21:01 |
| mordred | greghaynes: so - now the real question is - how do I get that env var all they way from a job into the nodepool builder running in that job | 21:01 |
| clarkb | mordred: you put in your your nodepool.yaml | 21:01 |
| cloudnull | thanks clarkb | 21:01 |
| clarkb | mordred: each image has an env section you can set | 21:01 |
| *** ijw has joined #openstack-infra | 21:01 | |
| mordred | clarkb: oh - right | 21:02 |
| clarkb | cloudnull: 9eadc9db-a4d5-4399-bdbd-79097450dfa7 and 3b225e12-0083-470f-b17a-cc0d06ac4484 | 21:02 |
| *** annegentle has quit IRC | 21:03 | |
| *** ijw has quit IRC | 21:03 | |
| clarkb | cloudnull: this second pair is from the logs I linked you earlier and the first pair from the one I just linked | 21:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache https://review.openstack.org/378999 | 21:03 |
| *** rtheis has quit IRC | 21:03 | |
| mordred | clarkb: like that ^^ ? | 21:03 |
| clarkb | cloudnull: in both cases ssh between instances on their private IPs seems to have timed out/failed. Our security groups for those nodes should be wide open and we allow all traffic between instances in our iptables setup (but I will work on confirming that) | 21:03 |
| *** pvaneck has quit IRC | 21:04 | |
| *** tqtran_ has quit IRC | 21:04 | |
| *** spzala has joined #openstack-infra | 21:04 | |
| clarkb | mordred: ya I think thats it | 21:04 |
| *** tuannguyen has quit IRC | 21:04 | |
| mordred | greghaynes: also - is there a 'good' way to get a dib build to use a particular ubuntu mirror? | 21:04 |
| jlvillal | clarkb: In regards to MULTI_HOST https://review.openstack.org/#/c/352790/14/devstack-vm-gate.sh | 21:04 |
| clarkb | mordred: you might want to conditionally add it if the file exists | 21:04 |
| greghaynes | mordred: heh | 21:04 |
| clarkb | mordred: so that the devstack plugin works on not our jenkins hosts | 21:04 |
| greghaynes | mordred: DIB_DISTRIBUTION_MIRROR=foo is what I use, but theres caveats | 21:04 |
| jlvillal | clarkb: How about MULTI_NODE? IRONIC_MULTI_NODE? IRONIC_MULTI_HOST? | 21:04 |
| clarkb | if [ -f /path/to/file ] ; then append that line to file | 21:05 |
| clarkb | mordred: ^ | 21:05 |
| *** ijw has joined #openstack-infra | 21:05 | |
| *** ijw has quit IRC | 21:05 | |
| clarkb | jlvillal: Thats probably a better question for devstack devs. I just wanted to point out that already has a meaning in devstack and its not what ironic appears to be using it for | 21:05 |
| *** ijw has joined #openstack-infra | 21:05 | |
| *** annegentle has joined #openstack-infra | 21:05 | |
| jlvillal | clarkb: Thanks. I might go with IRONIC_MULTI_NODE. less confusion for people | 21:06 |
| *** ddieterly[away] is now known as ddieterly | 21:06 | |
| clarkb | jlvillal: ya thats pretty explicit, I like it | 21:06 |
| greghaynes | mordred: basically if you want to get fancy with /etc/hosts you need to also use apt-sources | 21:06 |
| greghaynes | mordred: er with /etc/apt/sources.list | 21:06 |
| *** yamahata has joined #openstack-infra | 21:06 | |
| *** baoli has joined #openstack-infra | 21:06 | |
| *** tqtran has joined #openstack-infra | 21:06 | |
| mordred | greghaynes: what if I just want to use the exact same setup in /etc/apt as my build host has? | 21:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache https://review.openstack.org/378999 | 21:07 |
| greghaynes | mordred: apt-sources lets you point at a file on your build host. If you use sources.d then youre out of luck unless you want to add some features | 21:07 |
| *** psilvad has quit IRC | 21:07 | |
| clarkb | cloudnull: ya our script definitely attempts to add iptables rules for v4 nad v6 to allow all traffic between the two hosts. Now to confirm it on some actual instances | 21:07 |
| *** e0ne has quit IRC | 21:07 | |
| mordred | clarkb: ^^ updated per your suggestion | 21:07 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case https://review.openstack.org/352790 | 21:07 |
| *** kzaitsev_mb has quit IRC | 21:08 | |
| mordred | greghaynes: yah - I mostly would just love for the dib build that nodepool does in its functional test to use the per-region mirror | 21:08 |
| mordred | greghaynes: so I guess DIB_DISTRIBUTION_MIRROR= is probably fine | 21:08 |
| greghaynes | mordred: ah. So I too would love for the dib functests to do that | 21:08 |
| mordred | greghaynes: \o/ | 21:08 |
| greghaynes | mordred: on a call right now but I should link you to some code | 21:09 |
| mordred | woot | 21:09 |
| *** mat128 is now known as mat128|afk | 21:11 | |
| *** jkilpatr has joined #openstack-infra | 21:11 | |
| clarkb | cloudnull: ya iptables and ip6tables look all happy so I think that we should be able to communicate there. And in fact in the first job logs I linked much earlier today it did work several times before the failure | 21:12 |
| clarkb | mordred: you need to use >> not > :) | 21:13 |
| *** dprince has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 21:15 |
| *** tuannguyen has joined #openstack-infra | 21:15 | |
| jeblair | clarkb: cool. i'll begin work on restarting nodepool now | 21:16 |
| mordred | clarkb: hahahahaha | 21:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache https://review.openstack.org/378999 | 21:16 |
| *** dprince has joined #openstack-infra | 21:17 | |
| mordred | clarkb: yay for code review | 21:17 |
| *** amitgandhinz has quit IRC | 21:17 | |
| *** tiswanso has quit IRC | 21:19 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for upstart https://review.openstack.org/378985 | 21:19 |
| *** krtaylor has quit IRC | 21:21 | |
| clarkb | pabelanger: is the input confirmation thing related to files existing? making sure I understand ^ | 21:21 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update localrc for ironic-multinode case https://review.openstack.org/352790 | 21:21 |
| clarkb | put another way why is systemd ok? | 21:21 |
| pabelanger | clarkb: ya, there is still a race condition between between when check if the files doesn't exist and when we actually run ssh-keygen. I am hoping yes n | ssh-keygen works in this instance | 21:22 |
| *** roxanaghe has joined #openstack-infra | 21:22 | |
| pabelanger | clarkb: otherwise, I have no idea how to fix this, aside from removing it from glean | 21:22 |
| pabelanger | that is what we are fighting right now | 21:22 |
| pabelanger | clarkb: I am not sure why systemd works, I think it is because glean runs much later in the boot process | 21:23 |
| pabelanger | but that is a guess | 21:23 |
| *** eharney has quit IRC | 21:23 | |
| *** tonytan4ever has joined #openstack-infra | 21:23 | |
| clarkb | pabelanger: also is the yes n required? | 21:24 |
| clarkb | I guess the prompt is for "overwrite file?" | 21:24 |
| pabelanger | clarkb: right, here is the console log of the failure: http://paste.openstack.org/show/583393/ | 21:25 |
| *** dprince has quit IRC | 21:25 | |
| *** edmondsw has quit IRC | 21:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:25 | |
| *** pots has joined #openstack-infra | 21:25 | |
| jeblair | i have a nodepool launcher and deleter running | 21:26 |
| clarkb | now we watch and wait to see if it helps? | 21:27 |
| *** sdague has quit IRC | 21:27 | |
| *** Goneri has joined #openstack-infra | 21:28 | |
| *** tonytan4ever has quit IRC | 21:28 | |
| *** hogepodge has joined #openstack-infra | 21:28 | |
| jeblair | clarkb: yeah, unfortunately we will have lost all the events that would have cleared out the 'ready' nodes | 21:29 |
| jeblair | so i'm deleting ones that are 'ready' for more than 1h to speed things up a bit | 21:29 |
| jeblair | will do the same for 'used' | 21:30 |
| pabelanger | is it possible STATSD_HOST wasn't exported for the new launcher / deleter? I have't seen graphite.o.o update yet | 21:30 |
| jeblair | pabelanger: very possible, i'm just running them from screen experimentally | 21:31 |
| pabelanger | okay cool | 21:31 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add support for extra elastic-search graph filter https://review.openstack.org/255585 | 21:31 |
| *** matt-borland has quit IRC | 21:32 | |
| *** Goneri has quit IRC | 21:33 | |
| jeblair | pabelanger: restarted with statsd | 21:33 |
| *** amitgandhinz has joined #openstack-infra | 21:34 | |
| pabelanger | clarkb: cool, I think I have it working: http://paste.openstack.org/show/583394/ | 21:34 |
| pabelanger | jeblair: thanks | 21:34 |
| *** yamahata has quit IRC | 21:35 | |
| pabelanger | Woah, 935 'in use' nodes | 21:35 |
| pabelanger | that's new :) | 21:35 |
| *** jheroux has quit IRC | 21:36 | |
| pabelanger | clarkb: going to collect some data to see if it is an issue with systemd too, but I haven't found any failures yet | 21:38 |
| clarkb | ok | 21:38 |
| openstackgerrit | Merged openstack-infra/project-config: Added additional (disk|s3500|s3700) to entries https://review.openstack.org/378215 | 21:40 |
| *** Goneri has joined #openstack-infra | 21:41 | |
| *** kzaitsev_mb has quit IRC | 21:41 | |
| *** rwsu has quit IRC | 21:43 | |
| *** kgiusti has left #openstack-infra | 21:44 | |
| *** mriedem has quit IRC | 21:44 | |
| *** spzala has quit IRC | 21:44 | |
| *** EricGonczer_ has quit IRC | 21:47 | |
| *** marst has quit IRC | 21:50 | |
| *** rhallisey has quit IRC | 21:51 | |
| *** mdrabe has quit IRC | 21:51 | |
| *** mdrabe has joined #openstack-infra | 21:51 | |
| jeblair | clarkb, pabelanger, fungi: the 2 nodepool processes are taking approx equal cpu time now (i think this is good) | 21:51 |
| *** tuannguyen has quit IRC | 21:51 | |
| clarkb | I concur that that is a good thing | 21:52 |
| *** tuannguyen has joined #openstack-infra | 21:52 | |
| pabelanger | agreed | 21:52 |
| fungi | that's much more of a balance than i anticipated | 21:52 |
| *** Goneri has quit IRC | 21:53 | |
| *** baoli has quit IRC | 21:53 | |
| pabelanger | and we are continuing to grow on used nodes, 1033 now | 21:54 |
| jeblair | now that they are both cpu bound, we can probably *start* to draw conclusions about how the system is behaving (but we still have a bunch of leaked nodes from the restart) | 21:54 |
| *** rajinir has quit IRC | 21:55 | |
| jeblair | pabelanger: and yeah, that's probably a good sign | 21:55 |
| fungi | i suppose if we moved the db into trove we could even divide these between physical servers now? | 21:56 |
| jeblair | fungi: yes, but the host isn't cpu bound yet, so not urgent | 21:57 |
| *** tuannguyen has quit IRC | 21:57 | |
| *** hasharAway has quit IRC | 21:57 | |
| jeblair | fungi: i think i fall on the side of lets do that when we change the launchers to zk | 21:57 |
| pabelanger | clarkb: this just happened with ubuntu-xenial in my testing: http://paste.openstack.org/raw/583397/ | 21:57 |
| jeblair | for now at least :) | 21:57 |
| pabelanger | clarkb: first time I have seen that | 21:57 |
| fungi | oh, sure. was mostly pontificating | 21:57 |
| clarkb | pabelanger: is it formatted like that on your end you mean? | 21:58 |
| pabelanger | clarkb: no, that glean is hung on lo0 configuration, and I don't think we have setup a timeout | 21:58 |
| *** baoli_ has joined #openstack-infra | 21:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:58 | |
| pabelanger | clarkb: I guess we need to add TimeoutStartSec to systemd | 21:59 |
| clarkb | I thought glean had its own internal timeout | 22:00 |
| clarkb | also lo should just come up as its logical anyways | 22:00 |
| pabelanger | I'll let the server run and see if it timesout | 22:00 |
| *** jamesdenton has quit IRC | 22:01 | |
| *** thcipriani has quit IRC | 22:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Make docs non-voting for quark, remove template https://review.openstack.org/377908 | 22:02 |
| *** gordc has quit IRC | 22:02 | |
| *** jamesdenton has joined #openstack-infra | 22:03 | |
| *** kzaitsev_mb has quit IRC | 22:04 | |
| *** baoli_ has quit IRC | 22:04 | |
| *** thcipriani has joined #openstack-infra | 22:04 | |
| * mordred boggles at "hung on lo0 configuration" | 22:05 | |
| *** dtardivel has joined #openstack-infra | 22:05 | |
| pabelanger | Ya, we likely should impose some default timeout, otherwise, systemd keep trying to the end of time | 22:06 |
| pabelanger | A start job is running for Glean fo...erface lo (11min 26s / no limit) | 22:06 |
| greghaynes | oh right, I ran in to that and hacked in a fix and forgot to push it | 22:07 |
| *** marst has joined #openstack-infra | 22:07 | |
| greghaynes | yes put a timeout please | 22:07 |
| clarkb | I mean | 22:07 |
| greghaynes | its because systemd | 22:08 |
| clarkb | do you want your instance to come up with no lo interface? | 22:08 |
| mtreinish | haha, I like that >11min to bring up lo | 22:08 |
| greghaynes | if the options are dont come up at all or come up without lo then yes | 22:08 |
| *** tphummel has quit IRC | 22:08 | |
| clarkb | I want to say thats a recipe for broken. Figuring out why it isn't working and correcting that is what we should probably focus on | 22:08 |
| greghaynes | this also fails in other cases | 22:08 |
| greghaynes | think dhcp | 22:08 |
| greghaynes | or no connectivity | 22:09 |
| clarkb | lo isn't dhcp'd though | 22:09 |
| clarkb | and has no connectivity issues | 22:09 |
| greghaynes | right, you need the timeout for the other cases too though | 22:09 |
| *** mdrabe has quit IRC | 22:09 | |
| greghaynes | (which is where I hit it) | 22:09 |
| *** tqtran has quit IRC | 22:09 | |
| clarkb | sure we can timeout the other interfaces | 22:09 |
| pabelanger | Ya, I think there is something bigger going on here, something has blocked glean.sh I think | 22:09 |
| clarkb | but I think lo is a special case that should work 100% of the time | 22:09 |
| greghaynes | right, yea, different bug | 22:09 |
| pabelanger | and is related to ssh host keys I suspect | 22:09 |
| *** flepied has quit IRC | 22:10 | |
| *** lezbar has quit IRC | 22:10 | |
| pabelanger | going to update 378985 for systemd and test | 22:10 |
| *** krtaylor has joined #openstack-infra | 22:10 | |
| *** salv-orlando has quit IRC | 22:10 | |
| jeblair | pabelanger: the zuul job queue is probably more accurate than nodepool right now (until the leaked nodes are cleared out). it says we're running 855 jobs. which is a record. :) | 22:11 |
| *** lezbar has joined #openstack-infra | 22:11 | |
| *** tqtran has joined #openstack-infra | 22:11 | |
| *** Goneri has joined #openstack-infra | 22:11 | |
| pabelanger | come on 2.1 kilo-jobs per hour | 22:12 |
| openstackgerrit | Merged openstack-infra/system-config: Adding user 'maxwell' to OpenStack ID Production https://review.openstack.org/378686 | 22:12 |
| *** jamesdenton has quit IRC | 22:12 | |
| pabelanger | jeblair: Ya, that's the most blue I have see in the 'test nodes' graph on status.o.o/zuul | 22:13 |
| pabelanger | exciting | 22:13 |
| *** inc0 has joined #openstack-infra | 22:13 | |
| *** thorst has quit IRC | 22:13 | |
| *** baoli has joined #openstack-infra | 22:13 | |
| *** thorst has joined #openstack-infra | 22:14 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:14 | |
| *** rfolco_ has joined #openstack-infra | 22:14 | |
| *** annegentle has quit IRC | 22:14 | |
| *** rfolco_ has quit IRC | 22:14 | |
| *** baoli has quit IRC | 22:16 | |
| *** megm has quit IRC | 22:16 | |
| greghaynes | mordred: so last time I hacked on using the per-env mirrors I created https://review.openstack.org/#/c/279691/ | 22:17 |
| *** thorst has quit IRC | 22:18 | |
| greghaynes | mordred: which is only for trusty but the idea is we can make an element which has all the logic for the various distros | 22:18 |
| *** Guest92 has quit IRC | 22:18 | |
| *** esberglu has quit IRC | 22:19 | |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts https://review.openstack.org/378985 | 22:19 |
| *** megm has joined #openstack-infra | 22:19 | |
| greghaynes | mordred: I suspect youll run in to that exact issue though - the fact that we use a lot of the same env vars regardless of distro means you cant simply set a bunch of env vars and expect them to all DTRT | 22:20 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: php5-fpm 503 errors https://review.openstack.org/379022 | 22:20 |
| *** kzaitsev_mb has quit IRC | 22:21 | |
| *** amitgandhinz has quit IRC | 22:22 | |
| *** inc0 has quit IRC | 22:23 | |
| *** rlandy has quit IRC | 22:23 | |
| *** tonytan4ever has joined #openstack-infra | 22:23 | |
| clarkb | cloudnull: any luck with those uuids? let me know if you need a larger sample I can go trolling logstash for more instances | 22:24 |
| mordred | greghaynes: there are times when I wish dib config wasn't all through env vars | 22:26 |
| greghaynes | mordred: yeeeep | 22:26 |
| greghaynes | mordred: so I have a thing on my todo list | 22:26 |
| greghaynes | mordred: I want dib to accept a yaml file which is basically the diskimages: portion of nodepool.yaml | 22:26 |
| greghaynes | and you can say dib-build image-name | 22:26 |
| greghaynes | which alleviates some of the env var pain | 22:26 |
| jeblair | greghaynes: yes please and thank you | 22:26 |
| mordred | ++ | 22:27 |
| mordred | that would be ossum | 22:27 |
| greghaynes | :) | 22:27 |
| *** Apoorva_ has quit IRC | 22:27 | |
| mordred | that said - you know 12-factor apps are all configured via env vars, yeah? | 22:27 |
| pabelanger | I think python-tripleoclient wrote there own yaml to dib thing recently, see it in passing | 22:27 |
| mordred | https://12factor.net/config | 22:27 |
| greghaynes | haha | 22:27 |
| greghaynes | maybe we should all just use djb's envdir | 22:28 |
| *** Apoorva has joined #openstack-infra | 22:28 | |
| greghaynes | (not really) | 22:28 |
| mordred | factor 3 of 12 factor is the thing that makes me know that it's a pile of utter hogwash | 22:29 |
| mordred | "don't use config management, just write env vars" | 22:29 |
| greghaynes | its one of those things that sounds great until you actually do it with any sizeable app | 22:29 |
| mordred | yup | 22:30 |
| jesusaur | the underlying principle is still sound though: don't put deployment config in the same repo as the code | 22:31 |
| mordred | yes! | 22:32 |
| jesusaur | you just need to get fancier about where you put it | 22:32 |
| mordred | like- the part of the description that talks about not hard-coding config into source code ... | 22:32 |
| johnsom | greghaynes /home/jenkins/cache/files/get-pip.py wasn't found on the host: http://logs.openstack.org/01/379001/1/check/gate-neutron-lbaasv2-dsvm-scenario/dd421fe/console.html#_2016-09-28_22_05_47_902898 | 22:32 |
| mordred | that's very valid - and also I think makes it clear about the audience for the doc | 22:32 |
| jesusaur | ya | 22:32 |
| clarkb | biggest gerrit web ui peeve right now is this focus on the patchset | 22:32 |
| clarkb | when someone pushes a new patchset and I hit f5 I want to see the newpatchset! | 22:33 |
| phschwartz | clarkb: +100 | 22:33 |
| clarkb | but no, I have to manually change patchsets otherwise I get the old one again | 22:33 |
| *** thorst has joined #openstack-infra | 22:33 | |
| greghaynes | johnsom: hrmmm | 22:33 |
| phschwartz | That is very annoying. | 22:33 |
| clarkb | johnsom: greghaynes I think devstack-gate may copy it into devstack's files dir if using devstack | 22:34 |
| greghaynes | clarkb: any thoughts on what johnsom ran in to, I thought I was reading the element correctly... | 22:34 |
| clarkb | mordred: ^ you may need to account for that too | 22:34 |
| greghaynes | ah | 22:34 |
| clarkb | so its in /opt/stack/new/devstack/files I think | 22:34 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts https://review.openstack.org/378985 | 22:34 |
| mordred | clarkb: ah | 22:34 |
| *** esberglu has joined #openstack-infra | 22:34 | |
| johnsom | maybe /opt/stack/new/devstack/files/get-pip.py ? | 22:35 |
| *** cardeois has quit IRC | 22:36 | |
| *** trukise has quit IRC | 22:36 | |
| *** annegentle has joined #openstack-infra | 22:36 | |
| *** annegentle has quit IRC | 22:36 | |
| *** markvoelker has quit IRC | 22:36 | |
| clarkb | johnsom: yes I think so but check if thats where the devstack files dir is relative to devstack | 22:36 |
| clarkb | that path should be correct for the devstack repo itself | 22:36 |
| *** esberglu has quit IRC | 22:37 | |
| *** thorst has quit IRC | 22:38 | |
| clarkb | yup its devstack/files | 22:38 |
| *** thorst has joined #openstack-infra | 22:38 | |
| cloudnull | Clarkb sadly it's just been one of those days where I've been dragged away from computer. I'm currently relocating home and will be digging in there. | 22:41 |
| *** kzaitsev_mb has joined #openstack-infra | 22:42 | |
| cloudnull | Sorry for the delay. | 22:42 |
| clarkb | cloudnull: no problem. I am having similar issues with sick children | 22:42 |
| *** thorst has quit IRC | 22:43 | |
| *** tphummel has joined #openstack-infra | 22:44 | |
| *** ijw has quit IRC | 22:46 | |
| *** thorst has joined #openstack-infra | 22:48 | |
| pabelanger | clarkb: fungi: greghaynes: https://review.openstack.org/#/c/378985 is working properly now, updates to ssh-keygen | 22:48 |
| *** salv-orlando has joined #openstack-infra | 22:48 | |
| pabelanger | both under upstart and systemd | 22:48 |
| clarkb | jeblair: I deleted two 3 hour old nodepool instances in a used state. Checked their zuul launcher and they had both finished their jobs | 22:50 |
| clarkb | the next oldest instances are less than 1.5 hours old so just gonna let those age a bit more to simplify knowing what is and isn't safe to delete | 22:51 |
| *** tonytan4ever has quit IRC | 22:51 | |
| *** nadya has joined #openstack-infra | 22:52 | |
| *** salv-orlando has quit IRC | 22:52 | |
| *** thorst has quit IRC | 22:52 | |
| fungi | smarcet: have you ruled out the possibility that your 503 errors are due to apache not being tuned to handle the request volume you're getting? | 22:53 |
| *** Swami has quit IRC | 22:53 | |
| fungi | smarcet: jpmaxman: or are the apache-side errors just these "AH01075: Error dispatching request" and "AH01067: Failed to read FastCGI header" errors i'm seeing from proxy_fcgi in the log? | 22:54 |
| clarkb | rcarrillocruz: for when you wake you might want to respond to this UX survey results thing about quota management and plug your cloud launcher work. I think it would solve some of the problems they describe very well | 22:55 |
| *** sdake has quit IRC | 22:56 | |
| *** nadya has quit IRC | 22:57 | |
| fungi | smarcet: jpmaxman: oh, i do see some 503 errors in the apache access log, just not in the error log for some strange reason | 22:58 |
| *** ijw has joined #openstack-infra | 22:58 | |
| *** ddieterly has quit IRC | 22:59 | |
| jeblair | clarkb: yeah, we're at things newer than 1:20 are from the current invocation, so pretty close to that 1.5 hour mark | 22:59 |
| fungi | smarcet: jpmaxman: aha, error log simply doesn't mention the http response code. i see now correlating that a lot of them are for the same client ip (v6!) address | 23:00 |
| jeblair | clarkb: no ready nodes, which was the sign of a processing backlog. however, we have now worked through the zuul backlog, so things should be slowing down now | 23:01 |
| clarkb | jeblair: ya I think its caught up with work demands and so stuff is less pressurized | 23:01 |
| jeblair | clarkb, fungi, pabelanger: i *think* this worked well enough that it's worth writing the init scripts for real | 23:01 |
| pabelanger | ++ | 23:01 |
| fungi | jeblair: it's looking great so far | 23:01 |
| fungi | smarcet: jpmaxman: a quick grep of the current log shows 27 out of 31 of these 503 errors were associated with the same ipv6 address, 2 were with another v6 address in the same network, and 2 were from an ipv4 address | 23:03 |
| fungi | is that a load tester maybe? | 23:03 |
| *** tiswanso has joined #openstack-infra | 23:03 | |
| *** ijw has quit IRC | 23:03 | |
| *** smarcet has quit IRC | 23:03 | |
| *** tiswanso has quit IRC | 23:03 | |
| clarkb | fungi: does dns offer any insight? | 23:04 |
| *** tiswanso has joined #openstack-infra | 23:04 | |
| *** rwsu has joined #openstack-infra | 23:04 | |
| fungi | clarkb: no reverse dns, but the v6 addresses are all assigned to rackspace and the v4 address is at some provider in poland | 23:05 |
| pabelanger | jeblair: I can start work in init scripts in the morning | 23:05 |
| jeblair | pabelanger: that would be awesome, thanks! | 23:06 |
| *** hongbin has quit IRC | 23:10 | |
| *** rbrndt has quit IRC | 23:12 | |
| *** tqtran_ has joined #openstack-infra | 23:13 | |
| *** tqtran has quit IRC | 23:14 | |
| jeblair | Sep 28 23:10:28 zl01 puppet-user[17540]: Could not find data item zuul_launcher_keytab in any Hiera data file and no default supplied at /opt/system-config/ | 23:15 |
| jeblair | production/manifests/site.pp:923 on node zl01.openstack.org | 23:15 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts https://review.openstack.org/378985 | 23:16 |
| clarkb | cloudnull: looking at job logs for taht specific job we hae 44 successes none in osic. and 152 failures mostly in osic | 23:17 |
| *** kzaitsev_mb has quit IRC | 23:17 | |
| clarkb | given our instance distribution I would expect most failures and most successes in osic if this wasn't in some way related to the cloud/region | 23:17 |
| clarkb | but now I know that in general the setup can work. Just a matter of digging into why it isn't happy currently | 23:18 |
| jeblair | pabelanger, ianw: ^ it looks like the keytab is in a zuul-launcher hiera group, but the launchers are in the merger group | 23:18 |
| *** tqtran_ has quit IRC | 23:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 23:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 23:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 23:18 |
| jeblair | maybe it just needs to be moved in hiera | 23:19 |
| *** ijw has joined #openstack-infra | 23:19 | |
| pabelanger | jeblair: ah, I think set that up. Let me check what I did for puppet | 23:19 |
| pabelanger | Ya, I think we can just move it in hiera | 23:20 |
| jeblair | pabelanger: ok done :) | 23:20 |
| jeblair | cat zuul-launcher.yaml >>zuul-merger.yaml | 23:21 |
| jeblair | pabelanger: i just did that | 23:21 |
| jeblair | then deleted it | 23:21 |
| pabelanger | ack | 23:21 |
| *** Goneri has quit IRC | 23:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:22 | |
| *** ijw has quit IRC | 23:24 | |
| *** ijw has joined #openstack-infra | 23:29 | |
| *** sdake has joined #openstack-infra | 23:32 | |
| *** ijw has quit IRC | 23:34 | |
| *** sdake_ has joined #openstack-infra | 23:35 | |
| *** markvoelker has joined #openstack-infra | 23:37 | |
| *** hichihara has joined #openstack-infra | 23:37 | |
| *** sdake has quit IRC | 23:38 | |
| *** markvoelker has quit IRC | 23:41 | |
| jeblair | yay *now* we have jobs with afs stanzas | 23:42 |
| jeblair | now we need to merge a change to an infra repo :) | 23:42 |
| *** kzaitsev_mb has quit IRC | 23:43 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:44 | |
| *** tiswanso has quit IRC | 23:45 | |
| *** rwsu has quit IRC | 23:45 | |
| *** zhurong has joined #openstack-infra | 23:46 | |
| *** ijw has joined #openstack-infra | 23:47 | |
| mordred | \o/ | 23:47 |
| mordred | jeblair: got a link handy? | 23:48 |
| jeblair | mordred: to what? | 23:48 |
| pabelanger | rcarrillocruz: when you are online, it looks like the nodepool credentials for infracloud-chocolate are not correct | 23:48 |
| mordred | jeblair: oh - I misread your statement - now I understand it | 23:48 |
| mordred | :) | 23:49 |
| *** xarses has quit IRC | 23:51 | |
| pabelanger | rcarrillocruz: I see the issue, I think, typo in hieradata | 23:51 |
| pabelanger | fixing now | 23:52 |
| *** kzaitsev_mb has quit IRC | 23:52 | |
| *** amitgandhinz has joined #openstack-infra | 23:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 23:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 23:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 23:53 |
| *** ijw has quit IRC | 23:53 | |
| *** xyang1 has quit IRC | 23:54 | |
| openstackgerrit | Merged openstack-infra/system-config: Temporarily block port 80 and port 8080 on firehose https://review.openstack.org/378060 | 23:55 |
| jeblair | ooh a patch merged! | 23:56 |
| *** amitgandhinz has quit IRC | 23:58 | |
| *** roxanaghe has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!