| *** openstack has joined #openstack-infra | 15:46 | |
| anteaya | almost like watching an iguana try to ascend a wall | 15:46 |
|---|---|---|
| *** akshai has joined #openstack-infra | 15:46 | |
| *** asselin has quit IRC | 15:46 | |
| *** leifmadsen_ has joined #openstack-infra | 15:46 | |
| mordred | yolanda: yah - I just saw that and the mailing list thing | 15:46 |
| *** Sukhdev_ has joined #openstack-infra | 15:46 | |
| *** flaper87 has joined #openstack-infra | 15:46 | |
| *** ayoung has quit IRC | 15:47 | |
| *** rfolco has joined #openstack-infra | 15:47 | |
| *** rbrndt has joined #openstack-infra | 15:48 | |
| *** lezbar has joined #openstack-infra | 15:48 | |
| *** e0ne has joined #openstack-infra | 15:48 | |
| *** salv-orlando has joined #openstack-infra | 15:49 | |
| *** cfarquhar has joined #openstack-infra | 15:49 | |
| *** cfarquhar has quit IRC | 15:49 | |
| *** cfarquhar has joined #openstack-infra | 15:49 | |
| *** scheuran has quit IRC | 15:49 | |
| *** dims has joined #openstack-infra | 15:49 | |
| fungi | hrm, it looks like the bot made it through all its configured channels, so we're back in business (for a few minutes at least) | 15:50 |
| fungi | i'll give it a bit before i fire statusbot up, just so i don't freak out freenode | 15:50 |
| mordred | unfortunately, there is a fundamental deficiency in openstack that means that fully expressing what we need to here is complex client-side config | 15:50 |
| Kiall | quick review pls? Switches designate DSVM gate to use a DS plugin - https://review.openstack.org/#/c/250895/ | 15:50 |
| mtreinish | mordred: heh, that bug is even worse when you look at the openstack-health graphs: http://status.openstack.org/openstack-health/#/ | 15:50 |
| mtreinish | we're gonna have a giant peak today | 15:51 |
| *** electrofelix has joined #openstack-infra | 15:51 | |
| *** bpokorny has joined #openstack-infra | 15:51 | |
| mtreinish | fungi, mordred: if you get a sec I'm curious what the exact duration was for the subunit2sql migration. We should be writing it in the log file /var/log/subunit2sql_migration.log on logstash.o.o | 15:52 |
| *** jtomasek has joined #openstack-infra | 15:52 | |
| *** corvus is now known as jeblair | 15:52 | |
| *** bpokorny has quit IRC | 15:52 | |
| *** amotoki has quit IRC | 15:53 | |
| *** leifmadsen has quit IRC | 15:55 | |
| *** aspiers has joined #openstack-infra | 15:55 | |
| *** leifmadsen_ is now known as leifmadsen | 15:56 | |
| *** ajmiller has joined #openstack-infra | 15:56 | |
| *** jistr has quit IRC | 15:57 | |
| anteaya | November 30 third-party meeting logs: http://paste.openstack.org/show/480387/ | 15:57 |
| *** asselin has joined #openstack-infra | 15:58 | |
| *** jasondotstar has joined #openstack-infra | 15:58 | |
| anteaya | in the off change that can get saved in the channel log | 15:58 |
| *** sdague has joined #openstack-infra | 15:59 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Align story subscription stars on right https://review.openstack.org/250417 | 15:59 |
| *** mriedem has joined #openstack-infra | 16:00 | |
| openstackgerrit | Guido Günther proposed openstack-infra/jenkins-job-builder: Basic folder support https://review.openstack.org/134307 | 16:00 |
| *** e0ne has quit IRC | 16:02 | |
| *** amitgandhinz has joined #openstack-infra | 16:02 | |
| *** sabeen1 has joined #openstack-infra | 16:02 | |
| *** ayoung has joined #openstack-infra | 16:03 | |
| asselin | anteaya, no 3rd party meeting this morning (sorry I missed it, but don't see any logs either) | 16:03 |
| asselin | ? | 16:03 |
| mmedvede | asselin: meetbot is not working atm | 16:03 |
| mmedvede | the log - http://paste.openstack.org/show/480387/ | 16:03 |
| asselin | mmedvede, thanks | 16:04 |
| *** gordc has joined #openstack-infra | 16:05 | |
| *** btully has quit IRC | 16:05 | |
| anteaya | mmedvede: thank you :) | 16:06 |
| *** kevinbenton_ has quit IRC | 16:07 | |
| sdague | did the juno branch get deleted yet? | 16:07 |
| *** btully has joined #openstack-infra | 16:07 | |
| *** kevinbenton has joined #openstack-infra | 16:07 | |
| anteaya | asselin: and this thread as well, which discusses the current bot issues: http://lists.openstack.org/pipermail/openstack-dev/2015-November/080754.html | 16:08 |
| anteaya | sdague: morning | 16:08 |
| anteaya | sdague: I have not seen any activity consistent with the juno branch getting deleted | 16:08 |
| asselin | anteaya, mmedvede fyi there is a WIP zuul patch to work behind a proxy: https://review.openstack.org/#/c/240131/ | 16:08 |
| anteaya | asselin: thank you, I did not know about that | 16:09 |
| *** hashar has joined #openstack-infra | 16:09 | |
| mmedvede | aburmashev is not here, need to pass this info to him | 16:09 |
| asselin | anteaya, mmedvede watanabe isao is working on it from Japan. | 16:09 |
| anteaya | interesting | 16:09 |
| asselin | yes, seems like they're in a similar timezone | 16:10 |
| *** markvan_ is now known as markvan | 16:10 | |
| anteaya | oh perhaps they can help each other | 16:10 |
| asselin | yes | 16:10 |
| *** dizquierdo has joined #openstack-infra | 16:10 | |
| *** jordan_ has joined #openstack-infra | 16:11 | |
| *** swann has quit IRC | 16:11 | |
| anteaya | wonderful | 16:12 |
| *** tjones has joined #openstack-infra | 16:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Support serveraliases for https://trystack.org https://review.openstack.org/251454 | 16:15 |
| fungi | i've got statusbot rejoining channels now since meetbot seems to have remained stable for the past ~half hour | 16:15 |
| *** jordanP has quit IRC | 16:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add functional tests for boot from volume https://review.openstack.org/251150 | 16:15 |
| *** openstackstatus has joined #openstack-infra | 16:15 | |
| *** ChanServ sets mode: +v openstackstatus | 16:15 | |
| *** nadya_ has quit IRC | 16:15 | |
| anteaya | yay! | 16:15 |
| mordred | Shrews: let's see if that fixes the tests | 16:18 |
| Shrews | mordred: *nod* | 16:18 |
| anteaya | we have channel logs again | 16:18 |
| fungi | mordred: jeblair: did you see derekh's infra ml thread about 249351 breaking floating ip addresses in tripleo's cloud? | 16:18 |
| mordred | fungi: yes. | 16:18 |
| mordred | fungi: I have collected thoughts and am waiting for jeblair to be fully coffeed | 16:19 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add support to subunit2sql cli to specify a run_at time https://review.openstack.org/251456 | 16:19 |
| * mordred knows jeblair doesn't coffee in the morning - but whatever it is that he does to reach the state I think of as "coffeed" | 16:19 | |
| fungi | awesome. just making sure it didn't slip through the cracks. i'm still behind on e-mail | 16:19 |
| anteaya | I think he runs | 16:20 |
| *** hemnafk is now known as hemna | 16:20 | |
| *** jlanoux has joined #openstack-infra | 16:20 | |
| *** dangers_away is now known as dangers | 16:21 | |
| jeblair | asselin, zaro: do you know if the gearman plugin works with assigning jobs to executors running on the jenkins master? | 16:21 |
| asselin | jeblair, I've noticed it does not recently | 16:22 |
| asselin | jeblair, recently I believe is probably 6-8 months ago | 16:23 |
| d0ugal | Anyone got a moment to tell me if it looks like I've done this correctly? https://review.openstack.org/#/c/248772/ | 16:24 |
| *** pabelanger has joined #openstack-infra | 16:24 | |
| jeblair | asselin: yeah, looking in the logs, it should say 'build X on master with UUID' but i just see 'build X on with UUID', so i think maybe something has changed with the jenkins api | 16:25 |
| jeblair | zaro: ^ | 16:26 |
| pabelanger | oh look. People! Been sitting on the wrong side of a netsplit for a while :) | 16:26 |
| asselin | jeblair, yeah...I noticed it first when I installed newer ci systems which used the 'latest' jenkins available. I suspect if we pinned it to an older version it would work again. | 16:26 |
| openstackgerrit | Victor Ryzhenkin proposed openstack-infra/project-config: Add non-voting python34 jobs to murano-related projects https://review.openstack.org/251457 | 16:26 |
| pabelanger | fungi: when you have a moment. https://review.openstack.org/#/c/251454 Looks like we missed a server alias for trystack.org on SSL | 16:26 |
| *** ldnunes has joined #openstack-infra | 16:27 | |
| *** placko has quit IRC | 16:28 | |
| *** baoli has quit IRC | 16:29 | |
| fungi | pabelanger: that's not going to work. we'd need to get the static.openstack.org https cert reissued with those aliases added. i thought the old trystack.org server was going to host those redirects instead? | 16:31 |
| *** baoli has joined #openstack-infra | 16:31 | |
| *** aopincar has joined #openstack-infra | 16:32 | |
| dtantsur | dhellmann, hi, have a second for reno-related question/issue? | 16:33 |
| pabelanger | fungi: Not sure. I can go back to that team and figure out. But I suspect the server was / is going to be turned down | 16:34 |
| fungi | pabelanger: i'm also disinclined to reissue the static.o.o cert right now since sni for the list of domains on there costs around us$150 | 16:34 |
| fungi | we need to batch up additions/removals to make the cost a little more worthwhile | 16:34 |
| *** mwagner_lap has quit IRC | 16:35 | |
| *** mwagner_lap has joined #openstack-infra | 16:35 | |
| *** hdd has joined #openstack-infra | 16:35 | |
| anteaya | jroll: is someone in ironic telling third party ci systems to announce themselves on the mailing list? http://lists.openstack.org/pipermail/openstack-dev/2015-November/080806.html | 16:36 |
| pabelanger | fungi: true. I can bring the cost to the attention of management and see what they say. | 16:36 |
| anteaya | jroll: if yes, can we not do that please? http://lists.openstack.org/pipermail/openstack-dev/2015-November/080808.html | 16:36 |
| jeblair | fungi: mordred has caught me up on derekh's thing; i'm going to reply to the ml and work on a solution | 16:37 |
| fungi | pabelanger: yeah, as i predicted, the downside to doing sni on static.openstack.org is that we incur a relatively nontrivial monetary cost every time we want to alter the list of https domains hosted there | 16:37 |
| *** tjones has left #openstack-infra | 16:37 | |
| pabelanger | fungi: Ya. Let me atleast ask about cost in house. But, in general you would be for or against adding trystack.org and www.trystack.org into the static.o.o SSL cert? | 16:39 |
| notmorgan | fungi: at what point do we have enough certs for OpenStack that a proper CA is justified? | 16:39 |
| dstufft | Why does SNI cause you to need to reissue certificates | 16:39 |
| notmorgan | dstufft: they need to be aliases in the cert | 16:40 |
| fungi | dstufft: adding to the subjectaltname field | 16:40 |
| dstufft | SNI doesn't require SANs... | 16:40 |
| *** EricGonczer_ has quit IRC | 16:40 | |
| *** chlong has joined #openstack-infra | 16:40 | |
| dstufft | SNI lets you use completely different certificates based on hostname | 16:40 |
| *** Vivek_ is now known as Vivek | 16:40 | |
| *** Vivek has quit IRC | 16:40 | |
| *** Vivek has joined #openstack-infra | 16:40 | |
| notmorgan | fungi: or even maybe a wildcard *.o.o cert? [though i am not sure the state / support of those vs a CA] | 16:41 |
| fungi | dstufft: oh, that's useful. i suppose i could get a separate set of certificates for that particular addition | 16:41 |
| *** achanda_ has quit IRC | 16:41 | |
| dstufft | SAN is just the replacement to CN which allows multiple names instead of just th eone, SANs with a bunch of additional names are most useful when you need to deal with clients that don't support SNI | 16:41 |
| jeblair | (and vice-versa, SAN does not require SNI) | 16:41 |
| dstufft | (or when you only want to manage one certificate instead of multiple) | 16:42 |
| fungi | notmorgan: not a fan of wildcard certs. you basically reduce the security of all https sites for a domain to that of the least secure server using that cert | 16:42 |
| *** dizquierdo has quit IRC | 16:42 | |
| *** sdake has joined #openstack-infra | 16:42 | |
| notmorgan | fungi: yeah, oh man CAs are priiiicy | 16:42 |
| notmorgan | fungi: just looking at it. it would be nice if you could just issue for whatever o.o site you want though | 16:42 |
| dstufft | fungi: which CA are you using? | 16:43 |
| fungi | we host some less critical https sites on static.o.o so i wouldn't want to have a cert and key stored there which could be used to impersonate review.openstack.org for example | 16:43 |
| fungi | dstufft: namecheap | 16:43 |
| openstackgerrit | Merged openstack-infra/release-tools: Remove deprecated scripts https://review.openstack.org/249822 | 16:43 |
| fungi | pabelanger: so given that i can just buy a cert for a separate trystack.org redirect, that's one option. however that means either san or separate vhosts and not serveralias i think? | 16:44 |
| *** xyang1 has joined #openstack-infra | 16:45 | |
| dstufft | fungi: You might prefer something like StartSSL, no per certificate charge, you just pay yearly to get your class 2 validation | 16:45 |
| *** sdake_ has joined #openstack-infra | 16:46 | |
| dstufft | It's like $60 for an vlaidation adn you can generate unlimited certificates during the year that validation is active. | 16:46 |
| *** abregman has quit IRC | 16:46 | |
| *** sdake has quit IRC | 16:46 | |
| dstufft | (though you might need $60 for an individual validation and an additional $60 for the organization validation) | 16:46 |
| fungi | dstufft: thanks! that's definitely worth looking into. we have something like 20 subdomains we're doing https on at the moment | 16:47 |
| openstackgerrit | Victor Ryzhenkin proposed openstack-infra/project-config: Publish test results for murano tempest job https://review.openstack.org/251464 | 16:47 |
| *** sshnaidm has quit IRC | 16:47 | |
| fungi | i end up expensing a few hundred us$ annually for certificate renewals | 16:48 |
| *** baoli has quit IRC | 16:48 | |
| fungi | so certainly less than what you're saying they charge | 16:48 |
| dstufft | fungi: Note: StartSSL's web interface is kinda crummy, it uses HTTPS client certs and they do silly things like shut down for maintance over the weekend, but the savings can be pretty major since $60-$120 gives you unlimited DV certificates | 16:48 |
| *** EricGonczer_ has joined #openstack-infra | 16:48 | |
| fungi | and yeah, we're strictly dv (i can't imagine having to provide organizational paperwork to a ca for all of our subdomains) | 16:49 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Set the run_at time from the run's real start time https://review.openstack.org/251468 | 16:49 |
| dstufft | An extra $140 gets you validated for EV certificates, you don't have to provide paperwork for all the subdomains, just on the org itself | 16:49 |
| dstufft | but I wouldn't bother with EV tbh | 16:50 |
| fungi | nor i | 16:50 |
| openstackgerrit | Jean-Daniel Bonnetot proposed openstack-infra/project-config: Adding a new region on OVH cloud (BHS1) with a capacity of 80 ssd-osFoundation-3 https://review.openstack.org/251395 | 16:50 |
| dstufft | the only reason Python has EV certificates is because Digicert gave it to us for free | 16:50 |
| *** baoli has joined #openstack-infra | 16:50 | |
| mtreinish | jeblair, clarkb: on https://review.openstack.org/#/c/251468/1 do you have any ideas on where I can get the time from? | 16:50 |
| *** harlowja_at_home has quit IRC | 16:50 | |
| mtreinish | like, is it in the gearman event object or something? | 16:51 |
| jeblair | mtreinish: hrm, that'd probably be in jenkins | 16:51 |
| *** purp has quit IRC | 16:51 | |
| *** cinerama has joined #openstack-infra | 16:51 | |
| dstufft | fungi: Oh, the only other thing that StartSSL really charges for other than the validation that expires after a year, is if you ever need to revoke a certificate they charge like $30 for that | 16:52 |
| *** jpeeler has joined #openstack-infra | 16:52 | |
| *** jpeeler has joined #openstack-infra | 16:52 | |
| pabelanger | fungi: dstufft: Ya, I was looking into StartSSL to see if we would use it, looks like it might be possible | 16:52 |
| pabelanger | fungi: either way, I will defer to your preference for SSL certs :) | 16:53 |
| Clint | letsencrypt is probably about to drive down DV prices | 16:53 |
| *** kmartin has joined #openstack-infra | 16:53 | |
| jeblair | mtreinish: that's triggered by the zmq from jenkins, right? | 16:53 |
| pabelanger | Clint: Ya, working with that now for local stuff. Pretty awesome | 16:54 |
| mtreinish | jeblair: yeah, the same way the logstash side does it | 16:54 |
| *** purp has joined #openstack-infra | 16:54 | |
| *** armax has joined #openstack-infra | 16:54 | |
| openstackgerrit | Victor Ryzhenkin proposed openstack-infra/project-config: Add non-voting murano-congress integration job to congress repo https://review.openstack.org/251469 | 16:56 |
| jeblair | mtreinish: i don't think start time is included in the completed events. there are _started_ events, so you could remember the time from those and add it, or you might be able to update the zmq plugin to include start time. | 16:56 |
| jeblair | mtreinish: check with clarkb before proceeding. :) | 16:57 |
| jeblair | mtreinish: (another option, i don't like this one that much, but including it because brainstorming: use the jenkins api to look up the start time) | 16:57 |
| mtreinish | jeblair: yeah I was trying to avoid adding an additional query to it :) | 16:57 |
| mtreinish | because I was thinking I could also ask the log server for the last modified time which is probably good enough too | 16:58 |
| jeblair | mtreinish: i think that will be the end of the run, not the start | 16:59 |
| *** achanda has joined #openstack-infra | 16:59 | |
| mtreinish | the issue is right now we're adding the backlog of jobs from the subunit2sql migration and they all are showing today as the run+at time | 16:59 |
| jeblair | mtreinish: (so off by an hour) | 16:59 |
| mtreinish | jeblair: yeah it would be, but that's better than off by 6 days :) | 16:59 |
| *** shardy has quit IRC | 16:59 | |
| mtreinish | but I don't think thats the answer either | 16:59 |
| jeblair | mtreinish: if you want the end time, then you can just take the timestamp of the completed event from zmq and stick that in the gearman payload for the subunit worker | 16:59 |
| *** markvan has quit IRC | 16:59 | |
| dstufft | Clint: last I heard, several major CAs are planning on taking their DV certs to completely free as well in 2016/2017 | 17:00 |
| Clint | exactly | 17:01 |
| *** coolsvap has quit IRC | 17:01 | |
| *** ildikov has quit IRC | 17:01 | |
| *** markvan has joined #openstack-infra | 17:02 | |
| *** otter768 has joined #openstack-infra | 17:02 | |
| mtreinish | jeblair: start time is probably preferable, but in the grand scheme of things I don't think it matters too much, as long as it's consistent | 17:02 |
| jroll | anteaya: nobody is telling ironic third party folks to announce like that, that I know of. thingee is leading much of the onboarding, and we're generally pointing them at the general third party CI docs | 17:02 |
| *** jaosorior has quit IRC | 17:03 | |
| dstufft | Clint: It was bound to happen since CAs are basically always going to be in a race to the bottom | 17:04 |
| *** coolsvap has joined #openstack-infra | 17:05 | |
| *** derekh has joined #openstack-infra | 17:05 | |
| *** Sukhdev_ has quit IRC | 17:06 | |
| *** krtaylor has joined #openstack-infra | 17:06 | |
| *** sdake_ has quit IRC | 17:06 | |
| *** otter768 has quit IRC | 17:07 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Align story subscription stars on right https://review.openstack.org/250417 | 17:07 |
| anteaya | jroll: okay thanks | 17:08 |
| mordred | notmorgan: | 17:08 |
| mordred | gah | 17:08 |
| anteaya | jroll: hopefully redirecting this person is sufficient for the group | 17:08 |
| *** salv-orl_ has joined #openstack-infra | 17:08 | |
| jroll | anteaya: I hope so too :) we don't want to be different than other projects here | 17:09 |
| anteaya | jroll: thanks, you have no idea how lovely that is to hear :) | 17:10 |
| jeblair | fungi: email sent re tripleo nodepool thing | 17:10 |
| *** ashleighfarnham has joined #openstack-infra | 17:10 | |
| fungi | thanks! | 17:10 |
| *** electrical_ is now known as electrical | 17:11 | |
| anteaya | jroll: I also replied to your mid-cycle thread since the decision seems to be to do virtual | 17:11 |
| jroll | anteaya: :) | 17:11 |
| jroll | \o/ | 17:11 |
| notmorgan | lol | 17:11 |
| anteaya | jroll: having it listed on the virtual sprint page helps me | 17:11 |
| *** salv-orlando has quit IRC | 17:11 | |
| anteaya | jroll: if I read the thread correctly and you are in fact doing virtual | 17:12 |
| jroll | anteaya: I will do that, haven't made any efforts toward planning dates yet | 17:12 |
| jroll | yep | 17:12 |
| *** Sukhdev_ has joined #openstack-infra | 17:12 | |
| *** Sukhdev_ has quit IRC | 17:12 | |
| anteaya | jroll: okay thanks | 17:12 |
| anteaya | jroll: you could do a placeholder with dates to be determined | 17:12 |
| jroll | anteaya: I'm on vacation this week but will get it sorted, thanks for your reply :) | 17:12 |
| anteaya | jroll: oh sorry, happy vacation | 17:12 |
| anteaya | jroll: talk to you when you are back | 17:13 |
| anteaya | jroll: go offline | 17:13 |
| jroll | anteaya: cheers :) | 17:13 |
| anteaya | :) | 17:13 |
| *** eharney has joined #openstack-infra | 17:14 | |
| *** tmorin has left #openstack-infra | 17:14 | |
| *** boris-42_ has quit IRC | 17:15 | |
| *** Swami has joined #openstack-infra | 17:17 | |
| *** rebase has joined #openstack-infra | 17:19 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:21 | |
| derekh | jeblair: thanks for your reply, would it be possible to get the revert through as jobs are queuing up https://review.openstack.org/#/c/251438/ | 17:21 |
| *** Piet has joined #openstack-infra | 17:22 | |
| jeblair | derekh: yes, we're not using internap yet, so we can revert and restart | 17:24 |
| jeblair | derekh, fungi, mordred: ^ i approved the revert since i think we're all on the same page | 17:24 |
| derekh | jeblair: thanks | 17:25 |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:27 | |
| *** Guest23729 is now known as zeus | 17:27 | |
| fungi | sounds reasonable | 17:27 |
| *** zeus has quit IRC | 17:27 | |
| *** zeus has joined #openstack-infra | 17:27 | |
| *** ihrachys has quit IRC | 17:28 | |
| *** gyee has joined #openstack-infra | 17:28 | |
| *** thingee has joined #openstack-infra | 17:29 | |
| openstackgerrit | Merged openstack-infra/nodepool: Revert "Get public IPs from supplied networks" https://review.openstack.org/251438 | 17:30 |
| mordred | jeblair: ++ | 17:32 |
| lifeless | mordred: clarkb: have you reviewed the horizon 4-component pbr spec? | 17:34 |
| pleia2 | good morning | 17:36 |
| anteaya | morning pleia2 | 17:36 |
| *** jlanoux has quit IRC | 17:36 | |
| lifeless | mordred: clarkb: with your background supporting folk in the community and seeing things go wrong, I'd like your input, also on things that can still go wrong in infra with this | 17:36 |
| *** austin81 has joined #openstack-infra | 17:37 | |
| mordred | lifeless: I have not reviewed it - it's been on my todo list | 17:39 |
| mordred | lifeless: I'm not _crazy_ about hte idea of adding more flexibility to pbr | 17:39 |
| mordred | lifeless: nor about any of the xstatic packages | 17:39 |
| mordred | existing in the first place | 17:39 |
| mordred | BUT | 17:39 |
| *** unicell has quit IRC | 17:39 | |
| mordred | I'm not really offering other options to people, so I don't want to stand in the way | 17:39 |
| lifeless | mordred: so, please do review it from a minimisation of harm perspective then :) | 17:40 |
| mordred | kk | 17:40 |
| lifeless | mordred: one option, for instance, is to have the xstatic packages use setuptools_scm | 17:40 |
| mordred | ew | 17:40 |
| *** egonzalez has quit IRC | 17:40 | |
| mordred | that sounds worse | 17:40 |
| *** ifarkas has quit IRC | 17:41 | |
| mordred | krotscheck: I thought y'all were really close to being able to consume bower/npm stuff for horizon | 17:41 |
| lifeless | mordred: maybe | 17:41 |
| *** paul-carlton1 has joined #openstack-infra | 17:41 | |
| krotscheck | mordred: Define "really close". | 17:41 |
| krotscheck | if by "really close" you mean being asked to yackshave my way out of siberia, then yes. | 17:42 |
| lifeless | mordred: its actually very similar for versions, and does nothing else | 17:42 |
| *** Guest57402 has quit IRC | 17:42 | |
| lifeless | mordred: OTOH its then variation to deal with | 17:42 |
| lifeless | mordred: OTOOH these packages are not being hacked on or changed by definition so who cares? | 17:42 |
| krotscheck | mordred: Current sample tehreof, being asked a month ago, by fungi, to hold off on landing the pypi rename patches, which I was asked to do before landing an npm mirror, so that the wheel mirror patches can land first. | 17:42 |
| *** ildikov has joined #openstack-infra | 17:42 | |
| mordred | krotscheck: yak shaving is awesome | 17:43 |
| krotscheck | mordred: It's fun to make other people yackshave. | 17:43 |
| krotscheck | It's not fun to do it yourself. | 17:44 |
| *** edmondsw has joined #openstack-infra | 17:44 | |
| fungi | agreed. i get so much entertainment out of asking people to coordinate major modifications to the operation of a particular resource | 17:44 |
| fungi | the qa team were more eager to see us solve the wheel rebuilding problem than to see us solve javascript testing, hence my seemingly arbitrary prioritization between the two | 17:46 |
| openstackgerrit | Dong Ma proposed openstack-infra/jenkins-job-builder: Add support for cucumber reports plugin https://review.openstack.org/251489 | 17:47 |
| mtreinish | fungi: yes fixing the wheel rebuilding will save several min from devstack runs | 17:49 |
| *** markus_z has quit IRC | 17:50 | |
| *** hashar has quit IRC | 17:51 | |
| fungi | looks like 165240 is probably ready to go in now? | 17:53 |
| fungi | though i'm curious why the lone ? at the top of the global site manifest in the touchup 249969 change doesn't result in a syntax error | 17:54 |
| *** achanda has quit IRC | 17:54 | |
| fungi | krotscheck: ^ was that a typo on your part? | 17:54 |
| *** dimtruck is now known as zz_dimtruck | 17:55 | |
| krotscheck | fungi: Checking | 17:55 |
| krotscheck | fungi: Yep. | 17:56 |
| *** salv-orl_ has quit IRC | 17:56 | |
| *** ayoung has quit IRC | 17:57 | |
| *** salv-orlando has joined #openstack-infra | 17:57 | |
| krotscheck | fungi: Apologies if I seem grumpy about this, but when you asked me to hold off on the various mirror patches, you also stated that they'd be able to merge "in short order". | 17:58 |
| fungi | i understand, however that's what i believed given the current state the work and associated review feedback was in at the time | 17:59 |
| *** paul-carlton1 has quit IRC | 18:00 | |
| krotscheck | fungi: Would it be helpful if I add the mirror work (all of it) to the infra agenda so that it stays fresh in people's minds? | 18:00 |
| fungi | krotscheck: yep, it's worth discussing the coordination of those two changes tomorrow, and also would be good to make the coming differences more visible beyond the infra folk writing/reviewing it since this will touch things lots of our developer community see | 18:01 |
| *** salv-orlando has quit IRC | 18:01 | |
| *** gchamoul has quit IRC | 18:02 | |
| *** zz_dimtruck is now known as dimtruck | 18:02 | |
| *** derekh has quit IRC | 18:02 | |
| openstackgerrit | Merged openstack-infra/release-tools: Distinguish development milestones https://review.openstack.org/251377 | 18:02 |
| *** nadya_ has joined #openstack-infra | 18:03 | |
| *** aspiers has quit IRC | 18:03 | |
| *** rossella_s has quit IRC | 18:03 | |
| zaro | morning | 18:03 |
| pleia2 | hey zaro | 18:04 |
| fungi | zaro: any feel for whether we'll be in a good place tomorrow to attempt to reschedule the gerrit upgrade? | 18:05 |
| zaro | yeah i think so. | 18:06 |
| fungi | do we have an effective workaround/patch for the openid redirect problem? i've not been keeping a close enough eye on the progress there | 18:06 |
| zaro | the rollback script looks good. only issue left was the redirect problem but i think that's a non-blocker | 18:06 |
| *** gchamoul has joined #openstack-infra | 18:06 | |
| zaro | we probably have a few workarounds that we can discuss | 18:07 |
| *** shashank_hegde has joined #openstack-infra | 18:08 | |
| fungi | sounds good. thanks! | 18:08 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added additional pypi region hosts https://review.openstack.org/249969 | 18:08 |
| anteaya | zaro: can you line up the redirect workarounds on an etherpad so we can clearly compare them? | 18:09 |
| fungi | thanks krotscheck! +2 from me now | 18:09 |
| krotscheck | fungi: Danke | 18:09 |
| mordred | jeblair: I have communicated details about the networking things to the neutron folks | 18:09 |
| *** Volundr is now known as whoops | 18:09 | |
| jeblair | mordred: ++ | 18:10 |
| *** harlowja has joined #openstack-infra | 18:10 | |
| *** mikeit has joined #openstack-infra | 18:10 | |
| zaro | will do but it's these: https://review.openstack.org/#/c/248411/ and https://review.openstack.org/249714 | 18:10 |
| *** baoli_ has joined #openstack-infra | 18:10 | |
| *** chlong has quit IRC | 18:10 | |
| lifeless | mordred: clarkb: specifically, if you could review that spec *today* it would be most helpful to me in helping r1chardj0nes move forward | 18:11 |
| *** dtantsur is now known as dtantsur|afk | 18:11 | |
| *** edmondsw has quit IRC | 18:11 | |
| zaro | jeblair, asselin : which version of jenkins is having issues with gearman-plugin? | 18:11 |
| *** changbl has quit IRC | 18:12 | |
| asselin | zaro, definitely the most recent | 18:12 |
| asselin | zaro, and back a few months | 18:12 |
| *** paul-carlton1 has joined #openstack-infra | 18:12 | |
| anteaya | zaro: I'll make an etherpad with those two patches, if you can fill in detail | 18:12 |
| zaro | anteaya: i'm going to do a little more testing with rollback, just whether adding new user and project will work with rollback. other that that i think we are good with the testing. | 18:12 |
| krotscheck | mordred: As for "really close to consuming bower/npm", the answer is: We're now consuming npm things for dev build tools only in horizon. Bower is still on the todo list, since landing the ecosystem around npm takes priority. | 18:13 |
| *** amuller is now known as amuller_afk | 18:13 | |
| zaro | asselin: thanks. will look into that. | 18:13 |
| mordred | krotscheck: gotcha - thanks! | 18:13 |
| *** baoli has quit IRC | 18:14 | |
| *** changbl has joined #openstack-infra | 18:14 | |
| *** sridhar_ram has joined #openstack-infra | 18:14 | |
| *** adam_g has quit IRC | 18:14 | |
| *** adam_g has joined #openstack-infra | 18:14 | |
| *** adam_g has quit IRC | 18:15 | |
| *** adam_g has joined #openstack-infra | 18:15 | |
| mordred | jeblair, Shrews: so, TIL - router:external is not a flag to indicate external routing | 18:15 |
| mordred | jeblair, Shrews: router:external is a flag that indicates whether or not you can attach a router to that network | 18:15 |
| *** sabeen1 has quit IRC | 18:15 | |
| lifeless | mordred: wait, what ? | 18:15 |
| mordred | lifeless: yah | 18:15 |
| *** shashank_hegde has quit IRC | 18:16 | |
| *** unicell has joined #openstack-infra | 18:16 | |
| jeblair | mordred: if i stand with my head tilted like this, i can just barely see that. | 18:16 |
| mordred | lifeless: router:external is _not_ a flag that communicates that the network routes external traffic, although it does imply it since it does indicate ability to attach a router to the network | 18:16 |
| mordred | there is not currently a flag that indicates that a network routes packets externally | 18:16 |
| anteaya | zaro fungi etherpad with links to redirect workarounds and my limited description of the issue: https://etherpad.openstack.org/p/DXsJsLkUvw feel free to expand/correct | 18:16 |
| *** gnuoy has quit IRC | 18:16 | |
| *** jamespage has quit IRC | 18:17 | |
| jeblair | mordred: like "router? i don't have one on this network. but you can put one on here. it's external to me." | 18:17 |
| *** zeus has quit IRC | 18:17 | |
| *** flwang has quit IRC | 18:17 | |
| anteaya | zaro: okay great | 18:17 |
| mordred | jeblair: yah | 18:17 |
| *** trown is now known as trown|lunch | 18:17 | |
| jeblair | weird | 18:17 |
| *** anshprat has quit IRC | 18:17 | |
| *** rm_work has quit IRC | 18:17 | |
| mordred | jeblair: our request for a piece of metadata is understood | 18:17 |
| *** freyes has quit IRC | 18:18 | |
| *** calebb1 has quit IRC | 18:18 | |
| timrc | Feels good to be back :) | 18:18 |
| *** baoli_ has quit IRC | 18:18 | |
| timrc | Taking a week off from computerz was sadly more stressful than I thought it'd be. | 18:18 |
| *** timothyb89 has quit IRC | 18:19 | |
| *** odyssey4me has quit IRC | 18:19 | |
| *** timothyb89 has joined #openstack-infra | 18:19 | |
| notmorgan | timrc: oh hi. wb! | 18:20 |
| *** fungi has quit IRC | 18:20 | |
| *** dstufft has quit IRC | 18:20 | |
| zaro | anteaya: how about moving that info to https://etherpad.openstack.org/p/gerrit-2.11-upgrade ? | 18:20 |
| *** nadya_ has quit IRC | 18:21 | |
| zaro | anteaya: or https://etherpad.openstack.org/p/test-gerrit-2.11 | 18:21 |
| *** tbarron has quit IRC | 18:21 | |
| *** davidlenwell has quit IRC | 18:21 | |
| *** davideagnello has joined #openstack-infra | 18:22 | |
| anteaya | I can put it in the issues section of test-gerrit | 18:22 |
| *** zeus has joined #openstack-infra | 18:22 | |
| jeblair | infra-root: i went to restart nodepool to pick up the revert and encountered an error | 18:22 |
| jeblair | daemon.daemon.DaemonOSEnvironmentError: Unable to change process owner ([Errno 1] Operation not permitted) | 18:22 |
| anteaya | zaro: oh you are doing that | 18:22 |
| *** freyes has joined #openstack-infra | 18:22 | |
| *** zeus is now known as Guest21588 | 18:22 | |
| anteaya | zaro: even better | 18:22 |
| *** rm_work has joined #openstack-infra | 18:22 | |
| *** isviridov_away has quit IRC | 18:22 | |
| jeblair | phschwartz: i don't think that is related to your installation ^ | 18:23 |
| *** Guest21588 is now known as zeus` | 18:23 | |
| *** davidlenwell has joined #openstack-infra | 18:23 | |
| *** Guest83268 has quit IRC | 18:23 | |
| *** erikwilson has quit IRC | 18:23 | |
| *** bradm has quit IRC | 18:23 | |
| timrc | notmorgan: Hi! | 18:23 |
| jeblair | i wonder if python-daemon updated | 18:23 |
| *** krtaylor has quit IRC | 18:23 | |
| mordred | jeblair: that seems like fun | 18:23 |
| *** vaidy has quit IRC | 18:23 | |
| zaro | anteaya: actually it was already there. i'm just updating | 18:23 |
| jeblair | to be clear, nodepool is not currently running. | 18:23 |
| mordred | yah | 18:23 |
| jeblair | 2.1.0 uploaded on 2015-11-26 | 18:24 |
| mordred | jeblair: I have never seen that message myself | 18:24 |
| mordred | jeblair: AWESOME | 18:24 |
| mordred | jeblair: time for a downgrade and pin? | 18:24 |
| anteaya | zaro: ah sorry | 18:24 |
| jeblair | mordred: i think so, looking up revious version... (pypi should really have a "prev version" link) | 18:24 |
| *** reed has quit IRC | 18:24 | |
| mordred | jeblair: if only we knew someone who ran pypi :) | 18:25 |
| *** mrmartin has joined #openstack-infra | 18:25 | |
| *** jamespd has quit IRC | 18:25 | |
| *** jamespage has joined #openstack-infra | 18:25 | |
| phschwartz | jeblair: yeah, I just traced it back. It seems to be in that update | 18:25 |
| *** jamespage has quit IRC | 18:25 | |
| *** jamespage has joined #openstack-infra | 18:25 | |
| *** odyssey4me has joined #openstack-infra | 18:25 | |
| *** dstufft has joined #openstack-infra | 18:25 | |
| phschwartz | also, what size host do you suggest for nodepool? I am trying to build an image on a 40gb host that has just shy of 30gb open and I get a failure due to insufficient space | 18:26 |
| *** chlong has joined #openstack-infra | 18:26 | |
| *** sflanigan has quit IRC | 18:26 | |
| jeblair | i guess python-daemon 2.0.6 | 18:26 |
| *** gnuoy has joined #openstack-infra | 18:26 | |
| *** TheJulia has quit IRC | 18:26 | |
| *** pahuang has quit IRC | 18:26 | |
| *** Hunner has quit IRC | 18:27 | |
| *** crinkle has quit IRC | 18:27 | |
| *** yamamoto has quit IRC | 18:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Pin python-daemon to <2.1.0 https://review.openstack.org/251497 | 18:27 |
| *** erikmwilson has joined #openstack-infra | 18:27 | |
| *** mgagne has joined #openstack-infra | 18:28 | |
| *** mgagne is now known as Guest63453 | 18:28 | |
| *** doug-fish has quit IRC | 18:28 | |
| jeblair | i have installed 2.0.6 manually on nodepool and restarted | 18:28 |
| *** reed has joined #openstack-infra | 18:28 | |
| *** crinkle has joined #openstack-infra | 18:29 | |
| jeblair | phschwartz: jhesketh already fixed it in zuul https://review.openstack.org/250294 | 18:29 |
| anteaya | jeblair: did you see this patch? https://review.openstack.org/#/c/250294/ | 18:29 |
| Shrews | mordred: umm... wow | 18:29 |
| anteaya | ah so you did | 18:29 |
| phschwartz | jeblair: we need to get this out for nodepool too | 18:29 |
| jeblair | phschwartz: https://review.openstack.org/251497 | 18:30 |
| phschwartz | jeblair: any others that you can think of. I will update them if needed | 18:31 |
| *** bradm has joined #openstack-infra | 18:31 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Fix test attachments DB import https://review.openstack.org/251499 | 18:31 |
| *** jamespd has joined #openstack-infra | 18:31 | |
| *** sabeen has joined #openstack-infra | 18:32 | |
| *** calebb1 has joined #openstack-infra | 18:33 | |
| jeblair | phschwartz: we are using ~300G for 7 dib images | 18:33 |
| *** jcoufal has quit IRC | 18:33 | |
| *** paul-carlton1 has quit IRC | 18:33 | |
| *** fungi has joined #openstack-infra | 18:33 | |
| *** claudiub has quit IRC | 18:34 | |
| fungi | hrm, the freenode server into which i was connected died just a few minutes ago. presumably impact from the attacks is continuing | 18:34 |
| fungi | at least the meetbot logs look like its current connection is unaffected | 18:35 |
| jeblair | nodepool is running and tripleo cloud looks good | 18:35 |
| *** fitoduarte has joined #openstack-infra | 18:36 | |
| *** hogepodge has joined #openstack-infra | 18:37 | |
| jeblair | i'm performing the final rsync from dfw->nyj01 | 18:37 |
| *** jistr has joined #openstack-infra | 18:37 | |
| jeblair | or, rather, bhs1 | 18:37 |
| jeblair | so many mirrors | 18:37 |
| phschwartz | Is there a way to limit how much space nodepool uses to build an image? | 18:37 |
| *** flwang has joined #openstack-infra | 18:38 | |
| mordred | phschwartz: not really. it will use the amount of space it uses | 18:38 |
| *** Guest63453 has quit IRC | 18:38 | |
| *** Guest63453 has joined #openstack-infra | 18:38 | |
| *** sflanigan has joined #openstack-infra | 18:38 | |
| *** sflanigan has joined #openstack-infra | 18:38 | |
| *** jordan_ has quit IRC | 18:38 | |
| mordred | phschwartz: we have a 500G volume attached to nodepool.o.o | 18:38 |
| fungi | phschwartz: there is a trivial way to limit the space it uses, though you may not be pleased with the result | 18:39 |
| *** Guest63453 is now known as mgagne | 18:39 | |
| *** eil397 has joined #openstack-infra | 18:39 | |
| jeblair | puppet is re-enabled on bhs1 | 18:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add functional tests for boot from volume https://review.openstack.org/251150 | 18:40 |
| *** pahuang has joined #openstack-infra | 18:41 | |
| *** krtaylor has joined #openstack-infra | 18:41 | |
| jeblair | mordred, fungi: i think we might be ready for https://review.openstack.org/251395 ? | 18:41 |
| mordred | jeblair: awesome. +2 from me - go head whenver! | 18:41 |
| *** Hunner has joined #openstack-infra | 18:42 | |
| *** Hunner has quit IRC | 18:42 | |
| *** Hunner has joined #openstack-infra | 18:42 | |
| *** tbarron has joined #openstack-infra | 18:42 | |
| mordred | I've got an appointment in just a bit, when I get back, I'l land the next puppet apply patch | 18:42 |
| fungi | jeblair: awesome! | 18:42 |
| pleia2 | more clouds | 18:43 |
| openstackgerrit | Merged openstack-infra/project-config: Adding a new region on OVH cloud (BHS1) with a capacity of 80 ssd-osFoundation-3 https://review.openstack.org/251395 | 18:44 |
| openstackgerrit | Stephen Lowrie proposed openstack/os-testr: Change to always parsing classes from test_id https://review.openstack.org/246636 | 18:44 |
| jeblair | (that needs images uploaded, so will be just a bit before it's used) | 18:44 |
| *** shashank_hegde has joined #openstack-infra | 18:44 | |
| *** TheJulia has joined #openstack-infra | 18:46 | |
| *** calebb1 is now known as calebb | 18:46 | |
| fungi | popping out to run some late lunch/early dinner errands but will be back asap | 18:47 |
| pleia2 | fungi: enjoy | 18:48 |
| *** HeOS has quit IRC | 18:48 | |
| *** isviridov_away has joined #openstack-infra | 18:49 | |
| *** abregman has joined #openstack-infra | 18:49 | |
| krotscheck | I remember at one point someone was asking for a maven/java mirror. Does anyone remember who that was before I dive into the logs? | 18:50 |
| anteaya | I can understand how name-filter "Performance" is used on rax, how is name-filter "ssd-osFoundation-3" used? | 18:50 |
| anteaya | krotscheck: I have no memory of you asking for a maven/java mirror | 18:50 |
| *** mikeit has quit IRC | 18:51 | |
| krotscheck | anteaya: It wasn't me, it was someone else, and I was looped into the conversation because I was doing mirror things for a different language | 18:51 |
| anteaya | krotscheck: oh someone else asked about a maven/java mirror | 18:51 |
| *** shashank_hegde has quit IRC | 18:51 | |
| mordred | krotscheck: I also do not remember who it was, but I also remember someone asking for one | 18:52 |
| anteaya | krotscheck: in that case, I have no memory of anyone asking about a maven/java mirror | 18:52 |
| *** weshay_xchat is now known as weshay | 18:52 | |
| krotscheck | Well, into the losg I go! | 18:52 |
| *** achanda has joined #openstack-infra | 18:52 | |
| dprince | jeblair: TripleO CI jobs are running again. Thanks! | 18:52 |
| anteaya | dprince: yay | 18:52 |
| anteaya | mordred: how is name-filter: 'nodepool' used on bluebox? | 18:55 |
| *** Hunner has quit IRC | 18:55 | |
| *** TheJulia has quit IRC | 18:55 | |
| *** vaidy has joined #openstack-infra | 18:56 | |
| *** pvaneck has joined #openstack-infra | 18:56 | |
| mordred | anteaya: aroo? | 18:56 |
| electrofelix | is anyone aware of an elasticsearch outage? | 18:57 |
| *** salv-orlando has joined #openstack-infra | 18:57 | |
| *** markmcclain has joined #openstack-infra | 18:57 | |
| electrofelix | http://status.openstack.org/elastic-recheck/ suggests no indexing in the last week? | 18:58 |
| *** shashank_hegde has joined #openstack-infra | 18:59 | |
| mtreinish | mordred, fungi, nibalizer: I think the openstack health api server didn't pull in the latest subunit2sql release, which is causing some issues | 18:59 |
| mtreinish | if you get a sec can someone check what version is installed in the venv? | 18:59 |
| *** electrofelix has quit IRC | 18:59 | |
| mtreinish | hmm, I bet: http://git.openstack.org/cgit/openstack-infra/puppet-openstack_health/tree/manifests/api.pp#n36 doesn't pip install -U | 19:00 |
| lifeless | mordred: oh, so yes, I grok that | 19:00 |
| lifeless | mordred: I slightly mistook what you said | 19:00 |
| anteaya | mtreinish: in project-config/nodepool/nodepool.yaml bluebox has a name-filter in its config | 19:01 |
| anteaya | mordred: ^^ | 19:01 |
| anteaya | mtreinish: sorry tab complete fail | 19:01 |
| *** anshprat has joined #openstack-infra | 19:01 | |
| anteaya | mordred: I'm curious as to what name-filter does | 19:01 |
| *** Hunner has joined #openstack-infra | 19:01 | |
| *** Hunner has quit IRC | 19:01 | |
| *** Hunner has joined #openstack-infra | 19:01 | |
| anteaya | electrofelix: yes, I am, I am waiting for clarkb to arrive | 19:02 |
| anteaya | though nibalizer and I discussed it last friday | 19:02 |
| mordred | anteaya: ah - it limits the returned flavors from the cloud to those having the given string in its name | 19:02 |
| anteaya | mordred: thank you | 19:02 |
| *** otter768 has joined #openstack-infra | 19:03 | |
| *** TheJulia has joined #openstack-infra | 19:04 | |
| clarkb | anteaya I looked briefly with nibalizer friday night and it seemes fine? | 19:05 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: remove release_many.sh https://review.openstack.org/251510 | 19:05 |
| clarkb | anyways I am driving homw from beach so not useful for several hours | 19:05 |
| clarkb | /var/log/logstash on the workers to see what they are doing | 19:05 |
| *** sileht has quit IRC | 19:05 | |
| clarkb | and maybe bigdesk/elasticsearch head to see if ES is unhappy | 19:06 |
| *** otter768 has quit IRC | 19:08 | |
| *** jaosorior has joined #openstack-infra | 19:08 | |
| *** changbl has quit IRC | 19:09 | |
| *** sdake has joined #openstack-infra | 19:10 | |
| *** btully has quit IRC | 19:10 | |
| anteaya | clarkb: oh I didn't see that in logs | 19:11 |
| anteaya | clarkb: kibana serves nothing | 19:11 |
| anteaya | for me | 19:11 |
| mriedem | clarkb: looks like logstash has been dead since 11/24? http://status.openstack.org/elastic-recheck/index.html | 19:11 |
| anteaya | clarkb: drive safe | 19:11 |
| *** btully has joined #openstack-infra | 19:11 | |
| mriedem | oh, nvm | 19:11 |
| jesusaurus | anteaya: is there an issue with logstash/kibana? | 19:11 |
| mriedem | read scrollback :) | 19:11 |
| *** unicell1 has joined #openstack-infra | 19:12 | |
| *** trown|lunch is now known as trown | 19:12 | |
| *** dangers is now known as dangers_away | 19:13 | |
| sdague | mriedem: want to summarize? | 19:13 |
| jesusaurus | oh wow, "Delay in Elastic Search: Indexing behind by 162 hours" | 19:13 |
| mriedem | sdague: indexing has been dead since 11/24 | 19:13 |
| sdague | oh, fun :( | 19:14 |
| *** dangers_away has quit IRC | 19:14 | |
| sdague | that's a bummer, because I -1ed a review earlier today because of data I didn't find in there | 19:14 |
| *** unicell has quit IRC | 19:14 | |
| mriedem | heh | 19:14 |
| mriedem | yeah, was looking for http://logs.openstack.org/94/242594/2/check/gate-tempest-dsvm-nova-v20-api/085f243/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-11-30_18_03_48_003 | 19:14 |
| blogan | AJaeger: ping | 19:14 |
| mtreinish | mriedem: that's the same time we shutdown the subunit2sql worker | 19:14 |
| jaosorior | 19:15 | |
| *** jaosorior has quit IRC | 19:15 | |
| *** hashar has joined #openstack-infra | 19:15 | |
| jeblair | the logstash job queue is essentially empty now | 19:15 |
| *** kozhukalov_ has joined #openstack-infra | 19:16 | |
| mtreinish | jeblair: how many left in the subunit queue? :) | 19:17 |
| *** dangers_away has joined #openstack-infra | 19:17 | |
| jeblair | mtreinish: they use the same gearman server, right? | 19:17 |
| mtreinish | yeah | 19:17 |
| jesusaurus | jeblair: yes | 19:17 |
| sdague | jeblair: yeh, but there isn't content in the index | 19:18 |
| jeblair | http://grafana.openstack.org/dashboard/db/zuul-status?from=1448889547126&to=1448911147126 | 19:18 |
| sdague | that' indexing behind by X hours | 19:18 |
| sdague | is an elastic search query for most recent records | 19:18 |
| sdague | and grabs the last time stamp | 19:18 |
| jeblair | wow, the logstash workers have a lot of "Error: everything ok" style entries | 19:19 |
| *** salv-orlando has quit IRC | 19:19 | |
| *** salv-orlando has joined #openstack-infra | 19:20 | |
| *** mancdaz has quit IRC | 19:20 | |
| jeblair | "FilterException: Timeout reading from CRM114" | 19:21 |
| jeblair | that may be a problem | 19:21 |
| jeblair | it goes back to at least nov 24; unsure about earlier because that's the end of the log rotation | 19:21 |
| sdague | yeh | 19:21 |
| sdague | so we don't really ever use the crm114 data | 19:21 |
| mriedem | what version of logstash are we running? | 19:21 |
| sdague | if just turning it off solves a thing, there is little loss there | 19:22 |
| mriedem | https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/141 | 19:22 |
| jesusaurus | mriedem: 1.3.3 and will upgrade to 2.0 rsn | 19:22 |
| *** mancdaz has joined #openstack-infra | 19:22 | |
| mriedem | ok, ignore that issue then | 19:22 |
| *** ihrachys has joined #openstack-infra | 19:23 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/infra-specs: add "complete the reviewable release automation" https://review.openstack.org/245907 | 19:24 |
| *** mkoderer has quit IRC | 19:26 | |
| anteaya | the workers have to be shut off before logstash is off, else the need to be killed | 19:26 |
| anteaya | s/the/they | 19:26 |
| *** yamamoto has joined #openstack-infra | 19:28 | |
| jesusaurus | anteaya: the gearman workers? | 19:28 |
| nibalizer | anteaya: jesusaurus im about | 19:28 |
| jeblair | i think the crm114 is a red herring | 19:29 |
| *** mkoderer has joined #openstack-infra | 19:29 | |
| jeblair | (it might be a problem, but i think we're correctly continuing past it) | 19:29 |
| jesusaurus | jeblair: that FilterException is from the gearman workers, yeah? does the logstash indexer log have anything insightful? | 19:30 |
| jeblair | {:timestamp=>"2015-11-24T06:45:06.308000+0000", :message=>"Failed parsing date from field", :field=>"logdate", :value=>"2015-11-24 06:41:40.274488", :exception=>java.lang.IllegalArgumentException: Invalid format: "2015-11-24 06:41:40.274488", :level=>:warn} | 19:30 |
| jeblair | jesusaurus: ^ | 19:30 |
| anteaya | jesusaurus: yes the gearman workers for logstash | 19:30 |
| *** Verilium has left #openstack-infra | 19:31 | |
| anteaya | jesusaurus: they are hard to get them to come down if logstash is offlined while they are still up | 19:31 |
| *** _ody has joined #openstack-infra | 19:31 | |
| jesusaurus | anteaya: oh true | 19:32 |
| anteaya | jesusaurus: small point, just sharing | 19:32 |
| jeblair | it's worth noting that also shows up on nov 23, before the restart | 19:32 |
| jesusaurus | jeblair: so a log has a timestamp that we aren't parsing correctly, but that wouldn't cause the slow down unless they were being retried or something | 19:33 |
| *** yamamoto has quit IRC | 19:33 | |
| jeblair | but those are the only log messages other than the restart. there are 6985003 instances of that. | 19:33 |
| jeblair | jesusaurus: are we sure it's a slowdown? | 19:33 |
| jeblair | jesusaurus, sdague: are we seeing the tail end move? | 19:33 |
| jesusaurus | nope | 19:33 |
| jesusaurus | not sure | 19:34 |
| jeblair | jesusaurus: is it possible that message means no log entries are being stored in the cluster because logstash can't parse the timestamp on any of them? | 19:34 |
| sdague | jeblair: I have no idea if the tail is moving, at 162 hours, I doubt it is | 19:35 |
| sdague | I suspect the whole pipeline is down | 19:35 |
| jeblair | sdague: well, i'm working my way through it, and so far i think the processers are working, logstash is questionable (date parsing error?), and i'm moving on to es now. | 19:36 |
| jesusaurus | jeblair: ohh, yeah, if all the logs changed (did oslo.log change or something?) then they would all be failing to parse and they would be inserted with no timestamp value | 19:36 |
| sdague | the swift logs often have dates that aren't parsable | 19:36 |
| jeblair | corvus@fuligin:~$ ssh elasticsearch02.openstack.org | 19:36 |
| jeblair | -bash: cannot create temp file for here-document: No space left on device | 19:36 |
| jeblair | this is not promising | 19:36 |
| sdague | yeh, that seems like a root cause :) | 19:36 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=1657&rra_id=2&view_type=&graph_start=1448307432&graph_end=1448912232 | 19:37 |
| jeblair | only 02 looks like that | 19:37 |
| *** amuller_afk is now known as amuller | 19:38 | |
| jeblair | "status" : "green", | 19:38 |
| jeblair | "number_of_data_nodes" : 6, | 19:38 |
| jeblair | es looks good, despite one of the nodes being out of space on / | 19:38 |
| jeblair | (/ is not the data volume) | 19:38 |
| jesusaurus | that is /, isn't the data on a separate block device? that looks like there is probably a log file explosion or similar to me | 19:39 |
| *** dangers_away is now known as dangers | 19:39 | |
| *** tsg has joined #openstack-infra | 19:39 | |
| nibalizer | /var/lib/elasticsearch has plenty of free space | 19:39 |
| nibalizer | (on es02) | 19:39 |
| jeblair | wow, it logs queries it can't fulfill | 19:39 |
| *** dangers has quit IRC | 19:39 | |
| sdague | jeblair: interesting | 19:40 |
| jeblair | org.elasticsearch.search.query.QueryPhaseExecutionException: [logstash-2015.11.26][0]: query[+((message:"error unknown environment py27 constraints") (message:"error unknown environm | 19:40 |
| jeblair | ent py34 constraints")) +(tags:console) +(project:openstack/neutron) +(build_queue:gate) +(build_queue:gate)],from[0],size[3000],sort[<custom:"@timestamp": org.elasticsearch.index.fi | 19:40 |
| jeblair | elddata.fieldcomparator.LongValuesComparatorSource@3b226247>!]: Query Failed [Failed to execute main query] | 19:40 |
| jeblair | Caused by: org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb] | 19:40 |
| jeblair | :) | 19:40 |
| *** HeOS has joined #openstack-infra | 19:40 | |
| jeblair | (i don't think that's related to the problem, just interesting) | 19:40 |
| openstackgerrit | Anand Shanmugam proposed openstack-infra/project-config: Adding bandit job for cloudpulse https://review.openstack.org/251223 | 19:41 |
| *** markmcclain has quit IRC | 19:41 | |
| jeblair | though, that may actually be the cause of the / fillup | 19:41 |
| *** markmcclain has joined #openstack-infra | 19:41 | |
| clarkb | thats the problem | 19:41 |
| clarkb | we probably dont have working index deletes | 19:42 |
| *** abregman is now known as abregman|ZzZzZz | 19:42 | |
| nibalizer | it filled up with log files though | 19:42 |
| clarkb | we should have 10 for one prr day | 19:42 |
| jeblair | clarkb: you've jumped 2 steps ahead of me, can you catch me up? | 19:42 |
| nibalizer | e.g. 2.0Gelasticsearch.log.2015-11-28 | 19:42 |
| clarkb | jeblair our cluster can handle abput 10 days worth of logs | 19:42 |
| clarkb | if index deletiona fail (we just switched to a new tool to do that but still troggered by cron) then it has to load more in memory | 19:43 |
| jeblair | clarkb: do i need bigdesk to see the indexes? | 19:43 |
| *** sdake has quit IRC | 19:43 | |
| clarkb | and our data set is too big for that beyond abput 10-11 days | 19:43 |
| openstackgerrit | Merged openstack-infra/nodepool: Pin python-daemon to <2.1.0 https://review.openstack.org/251497 | 19:43 |
| *** abregman|ZzZzZz is now known as abregman | 19:43 | |
| clarkb | jeblair no there is a simple curl command to use check my users history | 19:43 |
| clarkb | on 02 | 19:43 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make Elastic Recheck Watch more reusable https://review.openstack.org/251515 | 19:44 |
| clarkb | the I think its curl -X DELETE http://localhost:9200/indexnr | 19:44 |
| clarkb | *indexname | 19:44 |
| jeblair | curl -XDELETE http://localhost:9200/logstash-2015.10.30/ | 19:44 |
| jeblair | like that | 19:44 |
| clarkb | or run the root cron command by hand | 19:44 |
| clarkb | ya | 19:44 |
| clarkb | and maybe see if it fails and debug that | 19:44 |
| jeblair | clarkb: DELETE seems like a strange way to show an index | 19:45 |
| jeblair | {"acknowledged":true} | 19:46 |
| jeblair | did i just tell es to delete the index for 11.24? | 19:46 |
| clarkb | oh I misunderstood delete is delete | 19:46 |
| ihrachys | hey folks. it would be great if we merge a devstack patch that makes devstack use generated config files for neutron: https://review.openstack.org/#/c/204722/ so that we can remove the files from the tree. | 19:47 |
| clarkb | typing from car on spotty connection fail | 19:47 |
| jeblair | clarkb: yeah, apparently that can be really dangerous. | 19:47 |
| jeblair | does anyone else know how to *show* the list of indexes in elasticsearch? | 19:47 |
| anteaya | I do not know how to show the list of indexes | 19:48 |
| clarkb | their api docs are pretty good but having a hard time pulling them up from here | 19:49 |
| anteaya | stackoverflow suggests: curl http://localhost:9200/_aliases http://stackoverflow.com/questions/17426521/list-all-indexes-on-elasticsearch-server | 19:49 |
| jeblair | anteaya: thanks, that appears to produce something like what i'd expect | 19:50 |
| jeblair | curl http://localhost:9200/_aliases?pretty=true | 19:50 |
| anteaya | yay! | 19:50 |
| jeblair | is even better | 19:50 |
| anteaya | wonderful | 19:50 |
| jeblair | there are 30 of them | 19:51 |
| jeblair | oldest is logstash-0015.11.20 | 19:51 |
| jeblair | logstash-0015.11.24 is still there (for the moment at least) | 19:51 |
| jesusaurus | that looks like its probably from a misparsed timestamp | 19:51 |
| jeblair | jesusaurus: oh good catch | 19:51 |
| nibalizer | its itereseting that we have logstash-2015-date and logstash 0015-date | 19:51 |
| jeblair | logstash-2015.11.11 | 19:51 |
| jeblair | is the oldest in this millenium | 19:51 |
| jeblair | there are 19 of those | 19:52 |
| jeblair | so still >10 | 19:52 |
| nibalizer | curl 'localhost:9200/_cat/indices?v' is a different view that I think is better | 19:52 |
| jeblair | nibalizer: ++ | 19:52 |
| nibalizer | the size of the things in the wrong milleium are very small | 19:52 |
| nibalizer | 109.7gb vs 222.1kb | 19:53 |
| jeblair | https://etherpad.openstack.org/p/es-issues | 19:53 |
| jeblair | for those not on host | 19:53 |
| jeblair | i'm going to resort that table | 19:53 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/system-config: Add more logstash timestamp formats https://review.openstack.org/251518 | 19:54 |
| jesusaurus | jeblair: ^^ for the timestamp parsing | 19:54 |
| jeblair | jesusaurus: thanks! | 19:54 |
| jeblair | logstash-2015.11.24 is gone from that table | 19:54 |
| nibalizer | we have 3 elasticsearch.log files in /var/log/elasticsearch that are over 1G | 19:55 |
| nibalizer | which I think is why the disk is full | 19:55 |
| sdague | jesusaurus: any idea what's generating the microsec logs? | 19:55 |
| jeblair | sdague, jesusaurus: maybe soon we can ask logstash? :) | 19:55 |
| sdague | because that could be an indication of a different config issue somewhere as well | 19:55 |
| sdague | jeblair: :) | 19:55 |
| sdague | jeblair: maybe, I'm not even sure how I'd query that :) | 19:56 |
| jeblair | nibalizer, clarkb: so i think i have experimentally verified that the curl -XDELETE method of index deletion works | 19:56 |
| nibalizer | awesome | 19:56 |
| jeblair | curator is teh new hotness for deleting indexes? | 19:56 |
| *** dimtruck is now known as zz_dimtruck | 19:56 | |
| jeblair | /var/log/curator.log is, naturally, empty, likely on account of the disk being full | 19:57 |
| jesusaurus | yeah, curator became the officially-supported script-of-choice for deleting old indices | 19:57 |
| jeblair | nibalizer: i'm going to run curator by hand, okay? | 19:58 |
| nibalizer | jeblair: ok! | 19:58 |
| nibalizer | "curator --logfile /var/log/curator.log delete indices --older-than ${keep_for_days} --time-unit days --timestring %Y.%m.%d" seems to be our cron | 19:58 |
| jeblair | yeah, i'm omitting the logfile portion | 19:59 |
| jeblair | it may be succeeding | 19:59 |
| nibalizer | neat | 19:59 |
| jeblair | oh nope | 19:59 |
| jeblair | 2015-11-30 19:59:05,161 ERROR Error deleting one or more indices. Run with --debug flag and/or check Elasticsearch logs for more information. | 19:59 |
| jeblair | 2015-11-30 19:59:05,161 ERROR Got a TIMEOUT response from Elasticsearch. Error message: HTTPConnectionPool(host=u'localhost', port=9200): Read timed out. (read timeout=30) | 19:59 |
| nibalizer | o.O | 19:59 |
| jeblair | it's apparently trying these in batches, so it's still running and failing | 20:00 |
| *** placko has joined #openstack-infra | 20:00 | |
| jeblair | when it's done, i'll add --debug | 20:00 |
| jesusaurus | I think you can add '--timeout 60' before 'delete' as well | 20:00 |
| jeblair | hrm, i think they succeeded despite the timeout | 20:02 |
| jeblair | (makes sense since that was a client-side timeout) | 20:02 |
| nibalizer | way down on indexes | 20:03 |
| nibalizer | jeblair: i think it succeeded as well | 20:03 |
| jeblair | unfortunately, now that i added the logfile back to the command line, it says there isn't anything to do, so i can't verify the hypothesis that the disk being full was preventing the logfile from being written which was preventing it from running. | 20:03 |
| nibalizer | you could move the 'days-to-keep' up by one day | 20:04 |
| nibalizer | retention of 9 days | 20:04 |
| jeblair | (interestingly, it prints "No indices matched provided args:" to the console even with that logfile arg) | 20:04 |
| *** shardy has joined #openstack-infra | 20:04 | |
| jeblair | hrm | 20:05 |
| jeblair | it ran and worked and logged. | 20:05 |
| jeblair | nibalizer: i think we're going to have to clean up the disk and see if the problem reappears | 20:06 |
| nibalizer | remove the hugest es logs you think? | 20:06 |
| jeblair | nibalizer: yeah | 20:06 |
| *** placko has quit IRC | 20:06 | |
| nibalizer | ok | 20:06 |
| *** mpryc has joined #openstack-infra | 20:06 | |
| nibalizer | i can do that | 20:06 |
| jeblair | kk | 20:06 |
| nibalizer | want me to try to save them somewhere or? | 20:06 |
| nibalizer | i suppose i could try to gzip | 20:06 |
| * nibalizer removes elasticsearch.log.2015-11-28 | 20:08 | |
| jeblair | sdague: i think this means that all the data are actually in es, and if you query now, it should look like everything is caught up... does that jive? | 20:09 |
| nibalizer | we now have 10% free disk | 20:09 |
| jeblair | sdague: (note that we now have the last 9 days of data except we're mysteriously missing november 24) | 20:09 |
| anteaya | kibana serves me a graph | 20:10 |
| anteaya | which is new | 20:10 |
| * jesusaurus sees a lot showing up in kibana now | 20:10 | |
| jeblair | syslog is logging again | 20:10 |
| jesusaurus | so I guess having too many indices prevented es from returning basic results? | 20:10 |
| jeblair | jesusaurus: yep | 20:11 |
| *** abregman is now known as abregman|afk | 20:11 | |
| *** zz_dimtruck is now known as dimtruck | 20:11 | |
| jesusaurus | jeblair: how much RAM do the data nodes have? | 20:11 |
| jeblair | jesusaurus: 60g | 20:12 |
| jeblair | the other es nodes are also logging the memory breaker error, but they are not logging tracebacks... | 20:13 |
| sdague | jeblair: yeh, that seems like what I'm seeing | 20:13 |
| sdague | so yes, looks like the data was there we just could get to it | 20:13 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-elastic_recheck: Add conf options for uncategorized.py https://review.openstack.org/251522 | 20:13 |
| jeblair | so i wonder if it's the tracebacks that cause es02 to have such larger logs | 20:13 |
| jeblair | and i wonder wy es02 is different in that respect? | 20:14 |
| sdague | when ER cycles again we'll know for sure | 20:14 |
| jeblair | sdague: yeah, and hopefully this time have logs from the event | 20:14 |
| *** doug-fish has joined #openstack-infra | 20:14 | |
| *** austin81 has left #openstack-infra | 20:16 | |
| jeblair | (is it because 02 is acting as a query coordinator or something?) | 20:16 |
| *** ldnunes has quit IRC | 20:16 | |
| *** freyes has left #openstack-infra | 20:17 | |
| *** hashar has quit IRC | 20:17 | |
| *** ayoung has joined #openstack-infra | 20:18 | |
| jesusaurus | jeblair: sdague: it looks like that's because es02 is the currently elected master | 20:18 |
| jeblair | oh that's at debug level | 20:18 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 20:18 |
| sdague | yeh, er just reindexed, it's now looking much better | 20:19 |
| sdague | http://status.openstack.org//elastic-recheck/ | 20:19 |
| jesusaurus | clarkb: do you think it would make sense to either bump the heap size a little or to bump the fielddata limit a little? | 20:19 |
| sdague | well, except that a grenade fail is top of the list | 20:19 |
| jeblair | sdague: we could have left well enough alone! | 20:20 |
| sdague | yeh, really :) | 20:20 |
| clarkb | jesusaurus we already bumped the limit the problem is we cant leak indexes with the size of our cluster | 20:20 |
| clarkb | if we delete them properly then no problem | 20:21 |
| jeblair | it looks like we're configured to log action.* at debug, so this matches our current logging config. so yeah, i'm assuming that the controller must be part of the action when running queries | 20:21 |
| *** tongli has joined #openstack-infra | 20:21 | |
| clarkb | even if we bump retention we still need working deletes | 20:21 |
| jeblair | we might want to consider changing that, or maybe changing it for action.search.type, but i don't know what else interesting might show up under that entry. | 20:22 |
| clarkb | otherwise we would have same fail | 20:22 |
| jesusaurus | clarkb: good point | 20:22 |
| *** eharney has quit IRC | 20:22 | |
| onovy | https://review.openstack.org/#/c/245603/ // some core for review one-liner pls? | 20:23 |
| * jeblair lunches | 20:23 | |
| *** ihrachys has quit IRC | 20:23 | |
| *** rfolco has quit IRC | 20:24 | |
| *** jistr has quit IRC | 20:27 | |
| *** sshnaidm has joined #openstack-infra | 20:29 | |
| *** achanda has quit IRC | 20:31 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-elastic_recheck: Add conf options for uncategorized.py https://review.openstack.org/251522 | 20:31 |
| *** amitgandhinz has quit IRC | 20:31 | |
| *** apoorvad has quit IRC | 20:32 | |
| anteaya | jesusaurus: have you a link for the patch when curator was added? | 20:33 |
| *** stevebaker has quit IRC | 20:33 | |
| mriedem | well i figured out what changed the timestamp format in the nova logs | 20:35 |
| *** salv-orlando has quit IRC | 20:35 | |
| *** jzerebec1i is now known as jzerebecki | 20:35 | |
| mriedem | https://review.openstack.org/#/c/241179/9/nova/utils.py | 20:35 |
| mriedem | merged on 11/24 | 20:35 |
| *** stevebaker has joined #openstack-infra | 20:35 | |
| mriedem | copied isotime() from oslo.utils but didn't keep the same timestamp format | 20:35 |
| jesusaurus | anteaya: https://review.openstack.org/#/c/222742/ | 20:35 |
| *** salv-orlando has joined #openstack-infra | 20:35 | |
| jesusaurus | mriedem: ah, good find | 20:36 |
| mriedem | i kind of want to revert that change... | 20:36 |
| anteaya | jesusaurus: thank you | 20:37 |
| mriedem | actually nvm, looks like it was a straight copy | 20:37 |
| *** eharney has joined #openstack-infra | 20:37 | |
| anteaya | jesusaurus: I understand why I didn't find it, I looked a puppet-logstash but this was merged on Oct. 1, but didn't become enabled until the system-config change which I didn't look at, thanks | 20:39 |
| anteaya | jesusaurus: I was looking at recent changes | 20:39 |
| anteaya | did you want to change the number of days curator keeps logs for from 10 to 9? | 20:42 |
| anteaya | that was mentioned, do we want that? | 20:43 |
| *** matrohon has joined #openstack-infra | 20:44 | |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Raise upper-constraints on python-cinderclient to 1.5.0 https://review.openstack.org/251535 | 20:45 |
| jesusaurus | nibalizer: do you think lowering the retention by a day will help? | 20:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for missing index on id columns in metadata tables https://review.openstack.org/251536 | 20:46 |
| jesusaurus | I don't think it'll make a difference because the cron was apparently failing to run | 20:47 |
| anteaya | jesusaurus: okay | 20:48 |
| anteaya | zaro: did you see yolanda has a -1 on https://review.openstack.org/#/c/241060/ | 20:49 |
| *** abregman|afk has quit IRC | 20:50 | |
| *** olaph has quit IRC | 20:50 | |
| *** doug-fish has quit IRC | 20:51 | |
| *** olaph has joined #openstack-infra | 20:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove py26 jobs from openstackclient https://review.openstack.org/249450 | 20:52 |
| lifeless | dims: reviewed | 20:53 |
| lifeless | dims: It would be wonderful if the test cases could be automated and included in the patchset, even if we're going to run them by hand | 20:53 |
| lifeless | dims: since this could easily regress | 20:53 |
| *** dims_ has joined #openstack-infra | 20:53 | |
| zaro | anteaya: thanks for the heads up. will fix. | 20:53 |
| *** doug-fish has joined #openstack-infra | 20:54 | |
| *** nadya_ has joined #openstack-infra | 20:55 | |
| *** doug-fis_ has joined #openstack-infra | 20:55 | |
| *** dims has quit IRC | 20:56 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for missing index on id columns in metadata tables https://review.openstack.org/251536 | 20:56 |
| *** nadya_ has quit IRC | 20:56 | |
| *** doug-fi__ has joined #openstack-infra | 20:57 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow setting the number of threads for indexing Gerrit changes https://review.openstack.org/241060 | 20:57 |
| bauzas | heya, reno is now in the novaclient tree, I'd love to get the CI job voting asap, in case someone could help me having Christmas gift just now https://review.openstack.org/#/c/248898/ :) | 20:58 |
| *** doug-fish has quit IRC | 20:58 | |
| *** sfinucan has joined #openstack-infra | 20:59 | |
| *** doug-fis_ has quit IRC | 21:00 | |
| *** tongli has quit IRC | 21:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-elasticsearch: Add logging configuration https://review.openstack.org/251540 | 21:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-elasticsearch: Stop logging actions at DEBUG level https://review.openstack.org/251541 | 21:00 |
| jeblair | clarkb, jesusaurus, nibalizer: ^ | 21:00 |
| anteaya | zaro: thanks | 21:00 |
| *** sarob_ has joined #openstack-infra | 21:01 | |
| anteaya | stevemar armax can you take a look at https://review.openstack.org/#/c/251195/ when you get a chance, tonyb's patch to add/update names and nicks for irc meetings | 21:01 |
| anteaya | stevemar armax I think your names should be in there, but up to you | 21:01 |
| * armax looks | 21:02 | |
| anteaya | armax: thanks | 21:02 |
| *** doug-fi__ has quit IRC | 21:02 | |
| stevemar | anteaya: looking | 21:02 |
| * mordred is back | 21:02 | |
| *** dims_ has quit IRC | 21:03 | |
| anteaya | stevemar: thank you | 21:03 |
| anteaya | mordred: whew, so glad you are back | 21:03 |
| tonyb | anteaya, stevemar, armax: comments very welcome. | 21:03 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-elastic_recheck: Add conf options for uncategorized.py https://review.openstack.org/251522 | 21:04 |
| tonyb | anteaya: I thought if I pushed it someone that was keen could run with it and they're more than 50% done. | 21:04 |
| tonyb | anteaya: the rest is easy but time comsuming. | 21:04 |
| *** otter768 has joined #openstack-infra | 21:04 | |
| *** Piet has quit IRC | 21:04 | |
| *** changbl has joined #openstack-infra | 21:04 | |
| anteaya | tonyb: yes, agreed, great sysphian work there | 21:04 |
| *** doug-fish has joined #openstack-infra | 21:04 | |
| jesusaurus | jeblair: lgtm | 21:05 |
| anteaya | jeblair: 251540 has an extra whitespace | 21:05 |
| jeblair | anteaya: i didn't write it. | 21:05 |
| * anteaya tries to not crowd jeblair's artistic vision | 21:05 | |
| anteaya | I believe you | 21:05 |
| jeblair | we should not fix it so that it is easier to diff from the upstream source | 21:06 |
| anteaya | okay | 21:06 |
| jeblair | i mean '"fix"' | 21:06 |
| anteaya | whitespace wins | 21:06 |
| *** kozhukalov_ has quit IRC | 21:07 | |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: add seperate job for monasca plugin in devstack install https://review.openstack.org/244893 | 21:07 |
| *** dprince has quit IRC | 21:07 | |
| *** hughhalf has left #openstack-infra | 21:08 | |
| *** otter768 has quit IRC | 21:08 | |
| *** doug-fish has quit IRC | 21:09 | |
| *** zeih has quit IRC | 21:09 | |
| *** weshay has quit IRC | 21:10 | |
| jeblair | BHS1 is spinning due to 'no valid host was found' errors; i sent ovh an email | 21:11 |
| *** [1]Thelo has joined #openstack-infra | 21:11 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for missing index on id columns in metadata tables https://review.openstack.org/251536 | 21:11 |
| ruagair | o/ | 21:13 |
| *** dangers_away has joined #openstack-infra | 21:13 | |
| *** dangers_away is now known as dangers | 21:13 | |
| *** [2]Thelo has quit IRC | 21:13 | |
| *** kozhukalov_ has joined #openstack-infra | 21:13 | |
| ruagair | anyone about ofay with public endpoints for OpenStackID? | 21:14 |
| *** dims has joined #openstack-infra | 21:14 | |
| ruagair | I can't get it to act as an OpenID endpoint for love or money | 21:14 |
| ruagair | (or for reading docs) | 21:14 |
| ruagair | I can, for example use https://login.launchpad.net/ and have users authenticate. I can't do the same with OpenStackID. | 21:15 |
| ruagair | Thoughts? | 21:15 |
| pleia2 | we're just using https://openstackid.org directly for zanata | 21:15 |
| fungi | ruagair: when you say reading docs, you mean the openstackid docs? | 21:16 |
| ruagair | I tried that. | 21:16 |
| fungi | or openid specification? | 21:16 |
| ruagair | Yes, I do fungi. I may have missed something though. | 21:16 |
| ruagair | Both :-) | 21:16 |
| ruagair | I've read a lot :-) | 21:16 |
| fungi | smarcet is apparently not in channel or i'd point you in his direction. i see he's awake and pushing reviews for openstackid though | 21:16 |
| ruagair | THanks pleia2. So at least I know that works. The problem must be some other incompatability. | 21:16 |
| fungi | he just updated the review for "OIDC - OpenId Connect Implementation" (198458) | 21:17 |
| pleia2 | ruagair: we definitely did have to do a bit of work (patch to openstackid) for support, cinerama did that work for us (but she's on other projects now, might not be available for help) | 21:17 |
| ruagair | oh, timely. | 21:17 |
| ruagair | THe apache mod claims openstackid is not a valid end point. | 21:18 |
| fungi | i believe he does at least subscribe to and reply on the openstack-infra@lists.openstack.org ml | 21:18 |
| ruagair | So at least I know I have the correct URL. | 21:18 |
| clarkb | ruagir thatd a known issue with diacovery | 21:18 |
| ruagair | Precisely clarkb | 21:18 |
| ruagair | oh, a known issue. | 21:19 |
| clarkb | openstackid is not spec compliant and requires and accepts header for the cml | 21:19 |
| clarkb | er xml | 21:19 |
| ruagair | ah. | 21:19 |
| clarkb | otherwise you get html landing page | 21:19 |
| ruagair | Well in that case, I might continue with login.launchpad.net and fixing OpenStackID could be a separate body of work. | 21:20 |
| fungi | yeah, openstackid i believe started out only implementing the openid bits necessary to support www.openstack.org's consumer implementation, and has been getting slowly expanded as we try to plug more things into it | 21:20 |
| tsg | clarkb: can you ok https://review.openstack.org/#/c/247936/ (we have sorted out the package availability issues in F22) | 21:20 |
| ruagair | (login.launchpad has it's own issues as well) | 21:20 |
| fungi | ruagair: that it does ;) | 21:20 |
| ruagair | login.launchpad works *great* for existing users I migrate. | 21:21 |
| ruagair | If you need to create a new login.launchpad account it won't return you :-) | 21:21 |
| *** mrmartin has quit IRC | 21:22 | |
| ruagair | Thanks for the heads up on OpenStackID :-) | 21:22 |
| *** kgiusti has quit IRC | 21:23 | |
| pleia2 | yeah, the "retirement" of launchpad in favor of ubuntu single sign on has been a big mess | 21:23 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for missing index on id columns in metadata tables https://review.openstack.org/251536 | 21:23 |
| pleia2 | with no communication | 21:23 |
| pleia2 | I have a whole pile of rants on the topic, but I'll keep them to myself :) | 21:23 |
| * Clint grins. | 21:23 | |
| *** jcooley has quit IRC | 21:23 | |
| ruagair | I'll lend you an ear pleia2 :-) | 21:24 |
| pleia2 | over beer! | 21:24 |
| *** alivigni has quit IRC | 21:24 | |
| *** amuller has quit IRC | 21:24 | |
| ruagair | oh, that might be a long wait. Let's drink in front of the screen and vent. | 21:24 |
| pleia2 | hehe | 21:24 |
| *** sdake has joined #openstack-infra | 21:25 | |
| *** jcooley has joined #openstack-infra | 21:25 | |
| *** mpryc has quit IRC | 21:26 | |
| anteaya | count down for stable team ptl nominations | 21:27 |
| *** aeng has quit IRC | 21:27 | |
| *** aeng has joined #openstack-infra | 21:29 | |
| *** dizquierdo has joined #openstack-infra | 21:29 | |
| *** doug-fish has joined #openstack-infra | 21:32 | |
| openstackgerrit | Ryan Hsu proposed openstack-infra/os-loganalyze: Whitelist .log.gz files from the passthrough https://review.openstack.org/217344 | 21:33 |
| *** aopincar has quit IRC | 21:34 | |
| anteaya | fungi: I am very confused about what this neutron plugin is trying to do, and I am uncertain their job actually does what they say it will do, this is the first patch which apparently was testing a dummy driver, I don't know why it was set up to do that: https://review.openstack.org/#/c/230358/9 | 21:34 |
| anteaya | and the patch I see today which is open: https://review.openstack.org/#/c/243725/2 | 21:34 |
| *** doug-fis_ has joined #openstack-infra | 21:35 | |
| anteaya | fungi: I find the whole situation odd, and would like to ask you to review if you could so that I can make sure this project is going forward as it should be | 21:35 |
| *** doug-fis_ has quit IRC | 21:35 | |
| anteaya | right now I don't have confidence it is | 21:35 |
| *** doug-fis_ has joined #openstack-infra | 21:35 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for missing index on id columns in metadata tables https://review.openstack.org/251536 | 21:35 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Fix indexes in db models https://review.openstack.org/251547 | 21:35 |
| *** doug-fish has quit IRC | 21:37 | |
| *** dizquierdo has quit IRC | 21:38 | |
| *** amitgandhinz has joined #openstack-infra | 21:38 | |
| fungi | anteaya: seems like maybe a microcosm of neutron itself. this bgpvpn plugin has multiple optional driver backends, one of which is bagpipe | 21:39 |
| fungi | so they set up a job to first test the networking-bgpvpn plugin to neutron, and now a job to test it with a specific (non-demonstration?) driver | 21:39 |
| fungi | export DEVSTACK_LOCAL_CONFIG+=$'\n'"NETWORKING_BGPVPN_DRIVER=BGPVPN:BaGPipe:networking_bgpvpn.neutron.services.service_drivers.bagpipe.bagpipe.BaGPipeBGPVPNDriver:default" | 21:40 |
| *** Piet has joined #openstack-infra | 21:41 | |
| fungi | and i guess it also needs the neutron networking-bagpipe driver along with that | 21:41 |
| anteaya | okay | 21:41 |
| anteaya | this is a first for me | 21:41 |
| anteaya | thanks for taking a look | 21:41 |
| fungi | doesn't strike me as unreasonable, but it's also possible i'm misreading | 21:42 |
| *** dims_ has joined #openstack-infra | 21:42 | |
| mtreinish | fungi: if you get a sec, could you check the subunit2sql version in the venv at /opt/openstack-health/.venv on health.o.o | 21:43 |
| *** baoli has joined #openstack-infra | 21:43 | |
| *** baoli has quit IRC | 21:43 | |
| *** dims has quit IRC | 21:43 | |
| anteaya | fungi: mostly I was confused | 21:44 |
| mtreinish | I have a feeling it didn't get updated to be >1.0.0 because pip isn't being run with -U | 21:44 |
| ruagair | A snag with Using Ubuntu login is that if you are a new user, you create an ubuntu account but Ubunut One does not return you. | 21:44 |
| * mtreinish should just change the requirements file | 21:44 | |
| ruagair | WE may need to wrap that in something and talk straight to the API. | 21:44 |
| anteaya | fungi: I'm also looking at sdague's comment here which was left after the first patch merged: https://review.openstack.org/#/c/230358/9/zuul/layout.yaml | 21:44 |
| openstackgerrit | Merged openstack-infra/project-config: Switch fuel-specs to openstack-specs-jobs template https://review.openstack.org/247471 | 21:46 |
| *** sdake_ has joined #openstack-infra | 21:47 | |
| *** sdake has quit IRC | 21:47 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for missing index on id columns in metadata tables https://review.openstack.org/251536 | 21:48 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Fix indexes in db models https://review.openstack.org/251547 | 21:48 |
| jhesketh | Morning | 21:50 |
| anteaya | morning jhesketh | 21:51 |
| anteaya | jhesketh: irc bots seem to have stablized currently, thanks with your work there yesterday | 21:51 |
| *** psanchez has quit IRC | 21:51 | |
| jhesketh | anteaya: oh good :-) | 21:52 |
| *** baoli has joined #openstack-infra | 21:52 | |
| *** psanchez has joined #openstack-infra | 21:53 | |
| fungi | mtreinish: "subunit2sql==0.12.0 # git sha 50ed21d" according to pbr freeze | 21:53 |
| *** fitoduarte has quit IRC | 21:54 | |
| *** matrohon has quit IRC | 21:55 | |
| fungi | anteaya: yeah, looks like networking-bgpvpn is one of those neutron projects in limbo which is chasing support for the most recent neutron release in their master branch rather than forking off a matching stable/liberty branch | 21:55 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Normalize volume objects https://review.openstack.org/251047 | 21:55 |
| Shrews | mordred: with the fixes ^^^^ | 21:56 |
| mordred | Shrews: woot! | 21:56 |
| fungi | pretty sure we've explained to them that this design is not expected to work, but they're doing suboptimal things there with the express knowledge of neutron leadership while they work on catching up to neutron master branch development and getting in sync | 21:56 |
| Shrews | though i now must exclude myself from voting | 21:56 |
| anteaya | fungi: yes this was the same project we talked about last week with the check requirements job | 21:56 |
| *** mwagner_lap has quit IRC | 21:57 | |
| *** sarob_ has quit IRC | 21:57 | |
| fungi | jhesketh: yes, thanks for banging your head against the bots and freenode's troubles. a complication you may have missed is that we explicitly pin chat.freenode.net to a specific ipv4 address in /etc/hosts on eavesdrop.o.o to work around a shortcoming in twisted's ipsec+ipv6 implementation | 21:58 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Make flavor-creation failure non-fatal https://review.openstack.org/251552 | 21:58 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Remove flavor creation https://review.openstack.org/251553 | 21:59 |
| *** mriedem is now known as mriedem_away | 22:00 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Normalize volume objects https://review.openstack.org/251047 | 22:01 |
| *** sdake_ has quit IRC | 22:02 | |
| fungi | jhesketh: and due to server move churn, maintenances and other events they often take addresses out of the official chat.f.n round-robin dns entries so from time to time we have to ssh into the server and manually tweak /etc/hosts to accommodate that | 22:02 |
| *** shardy has quit IRC | 22:03 | |
| fungi | i basically keep checking the round-robin and trying one of the couple of ipv4 addresses they still had in the rotation as alternatives in /etc/hosts to figure out which was behaving better, but in the end i still had to wait until at least one was behaving well enough to not drop/kick the bot and then start refusing connections again | 22:03 |
| fungi | looks like the attack must have lightened up because they've added a couple more addresses back to the rr now | 22:05 |
| *** notmyname has quit IRC | 22:05 | |
| *** thorst_ has quit IRC | 22:05 | |
| *** thorst_ has joined #openstack-infra | 22:06 | |
| *** dkranz has quit IRC | 22:06 | |
| *** notmyname has joined #openstack-infra | 22:06 | |
| *** markvoelker has joined #openstack-infra | 22:07 | |
| *** dkranz has joined #openstack-infra | 22:09 | |
| *** thorst_ has quit IRC | 22:10 | |
| anteaya | it looks to me like the puppet-apply job is starting to hit errors with puppetdb, several of them postgres errors: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22puppetdb%5C%22%20AND%20build_status:FAILURE%20AND%20message:%5C%22ERROR%5C%22 | 22:11 |
| anteaya | I'm I reading it correctly? | 22:11 |
| anteaya | motivation for my question, zaro's patch failed 2 of 3 puppet apply tests: https://review.openstack.org/#/c/241060/7 | 22:12 |
| anteaya | and I can't see why | 22:12 |
| tsg | lifeless, clarkb: wrt patch 247936, can you elaborate on "bindep"? | 22:13 |
| mtreinish | fungi: yeah that should be 1.0.1 | 22:13 |
| *** julim has quit IRC | 22:16 | |
| *** trown is now known as trown|outttypeww | 22:16 | |
| crinkle | anteaya: i think that error is just a result of the fact that puppet apply in noop mode won't actually create the 'puppetdb' role in postgresql, and so a resource that depends on it will see it as not there | 22:16 |
| *** robcresswell has quit IRC | 22:17 | |
| anteaya | crinkle: thank you for looking | 22:17 |
| crinkle | anteaya: but the apply test is smart enough to know it's just because it's in noop mode, which you can tell because it exits 0 for that bit http://logs.openstack.org/60/241060/7/check/gate-infra-puppet-apply-bare-precise/a481ff8/console.html#_2015-11-30_20_59_42_773 | 22:17 |
| anteaya | ah missed that thank you | 22:17 |
| jlk | Can somebody help me figure out what's stuck with this gerrit review? https://review.openstack.org/#/c/249949/ | 22:17 |
| anteaya | crinkle: any idea why the tests do fail? | 22:17 |
| jlk | looks like it has all the right +1s, except for jenkins | 22:18 |
| jlk | but all the gates passed | 22:18 |
| crinkle | anteaya: the real culprit is the template seems to be invalid http://logs.openstack.org/60/241060/7/check/gate-infra-puppet-apply-bare-precise/a481ff8/console.html#_2015-11-30_20_59_31_022 | 22:18 |
| *** robcresswell has joined #openstack-infra | 22:18 | |
| *** mpryc has joined #openstack-infra | 22:19 | |
| tsg | lifeless: when adding a new pkg to the CI slaves, enough to add it to "jenkins/data/bindep-fallback.txt"? | 22:19 |
| anteaya | crinkle: oh thank you | 22:20 |
| *** eharney has quit IRC | 22:20 | |
| anteaya | crinkle: I appreciate the education on how to navigate puppet-apply logs | 22:20 |
| tsg | lifeless: (particularly if there are some old distros the package may not be available on) | 22:20 |
| anteaya | crinkle: my gratitude | 22:20 |
| jeblair | jlk: it's in the queue at http://status.openstack.org/zuul/ | 22:22 |
| crinkle | anteaya: no problem :) | 22:22 |
| *** tiswanso has quit IRC | 22:22 | |
| jlk | jeblair: ah thanks. Didn't realize the queue was that long. | 22:22 |
| zaro | crinkle: thanks. how do you tell puppet to treat the number as an int? | 22:22 |
| docaedo | would any infra cores care to take a look at https://review.openstack.org/#/c/250061/ ? It's got a +2 already, but could use another look or two of course. | 22:23 |
| *** ZZelle has joined #openstack-infra | 22:23 | |
| zaro | other than what's there, $index_threads = 1 | 22:24 |
| lifeless | tsg: yep | 22:24 |
| lifeless | tsg: well, if we can't add it everywhere yet | 22:24 |
| lifeless | tsg: we won't be able to update requirements to versions that need it | 22:25 |
| asselin | anyone know how to make a job voting on a branch, but not on another? | 22:25 |
| lifeless | tsg: and itwould be really quite bad if the requirements job *passed* when thing will fail at a wider scale | 22:25 |
| lifeless | tsg: which a non-homogeneous setup would tend to encourage | 22:25 |
| anteaya | asselin: yes there are rules at the top of zuul/layout.yaml in project-config | 22:25 |
| anteaya | asselin: you should see plenty of examples there, let us know if you can't find what you are looking for | 22:26 |
| anteaya | asselin: if you search for voting: false you will find the section of the file where those rules live | 22:26 |
| crinkle | zaro: huh, it's kind of weird that it's not treating it as an int | 22:26 |
| fungi | mtreinish: looks like the puppet module wants to install latest from pip. any guesses as to why it doesn't? | 22:26 |
| *** mancdaz has quit IRC | 22:26 | |
| crinkle | zaro: you could do @index_threads.to_i > 1 | 22:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration for adding missing indexes https://review.openstack.org/251536 | 22:27 |
| asselin | anteaya, ok yeah, that's what I was doing. but now I understand better. default is true, so I just need to get the regex right. thanks! | 22:27 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Fix indexes in db models https://review.openstack.org/251547 | 22:27 |
| anteaya | asselin: welcome | 22:28 |
| mtreinish | fungi: I think it's the way we install requirements | 22:28 |
| *** mancdaz has joined #openstack-infra | 22:28 | |
| fungi | mtreinish: oh! i was looking at puppet-subunit2sql | 22:28 |
| fungi | i guess this is pulled in by openstack-health instead | 22:28 |
| fungi | my bad | 22:28 |
| mtreinish | fungi: I think http://git.openstack.org/cgit/openstack-infra/puppet-openstack_health/tree/manifests/api.pp#n36 just does pip install (without a -U) | 22:28 |
| fungi | indeed | 22:28 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow setting the number of threads for indexing Gerrit changes https://review.openstack.org/241060 | 22:29 |
| fungi | so, yes, expected behavior if you don't specify a higher minimum in requirements.txt | 22:29 |
| zaro | crinkle, anteaya ^ done, fingers crossed | 22:29 |
| mtreinish | fungi: I can change that to manually run pip with -U (it'll be an exec I don't think that module supports it with reqs files) | 22:30 |
| fungi | mtreinish: also potentially complicated if you're mixing some python packages from the distro | 22:30 |
| mtreinish | fungi: it shouldn't be because we venv it | 22:30 |
| fungi | oh, though this is in a virtualenv so presumably not | 22:30 |
| * anteaya crosses her fingers | 22:31 | |
| fungi | yeah, that | 22:31 |
| *** regXboi has quit IRC | 22:32 | |
| *** EricGonczer_ has quit IRC | 22:33 | |
| crinkle | zaro: i guess variables get translated into strings for templates, kind of surprising to me https://docs.puppetlabs.com/puppet/latest/reference/lang_template_erb.html#expression-printing-tags | 22:35 |
| *** lykinsbd has quit IRC | 22:35 | |
| anteaya | crinkle: what an interesting choice | 22:36 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-openstack_health: Make sure we install the latest requirements https://review.openstack.org/251564 | 22:37 |
| mtreinish | fungi: ^^^ I think I did that right | 22:37 |
| openstackgerrit | Pedro Alvarez Piedehierro proposed openstack-infra/storyboard: RFC: Add story_title to tag events. https://review.openstack.org/251566 | 22:37 |
| zaro | crinkle: wacko! but good to know. | 22:38 |
| zaro | thanks | 22:39 |
| *** aspiers has joined #openstack-infra | 22:39 | |
| crinkle | np | 22:39 |
| *** kozhukalov_ has quit IRC | 22:39 | |
| openstackgerrit | Merged openstack-infra/ciwatch: Use zuul gerrit event listener implementation https://review.openstack.org/248919 | 22:39 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Return Run.uuid in public methods https://review.openstack.org/251227 | 22:39 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add support to subunit2sql cli to specify a run_at time https://review.openstack.org/251456 | 22:40 |
| notmorgan | is there any reason we may have explicitly hidden pypi uploads? ( fungi, mordred, dhellmann, clarkb )? | 22:40 |
| dhellmann | notmorgan : ? | 22:41 |
| notmorgan | for example keystonemiddleware has the last 3 releases explicitly set to "hide" | 22:41 |
| dhellmann | notmorgan : I think that's the default behavior when a new release is uploaded | 22:41 |
| notmorgan | dhellmann: ksm has 2.4.0, 2.4.1, and 3.0.0 all set hidden | 22:41 |
| notmorgan | the old releases are supposed to be hidden | 22:41 |
| *** btully has quit IRC | 22:41 | |
| notmorgan | but 2.3.2 is the latest one showing up even though newer ones are uploaded | 22:41 |
| stevemar | dolphm: ^ | 22:41 |
| *** dkranz has quit IRC | 22:41 | |
| dhellmann | notmorgan : could it be the order they were uploaded? | 22:41 |
| stevemar | dhellmann: i brought this up last week if you recall | 22:41 |
| stevemar | dhellmann: nah, they were in that order | 22:42 |
| notmorgan | maybe? i can unhide the latest release if we want to try | 22:42 |
| anteaya | I don't think that is something we configure | 22:42 |
| notmorgan | i want to make sure i am not breaking things that were explicitly set | 22:42 |
| anteaya | if it is, I have never seen configuration for it | 22:42 |
| mordred | yah - we have not done that on purpose | 22:42 |
| mordred | infra-root: I'm going to land the next patch in the puppet-apply series - it should have zero effect on anything | 22:42 |
| dhellmann | anteaya : I believe we can tell pypi to hide or not hide when we upload but I don't know if we have support in our tools for that | 22:42 |
| stevemar | ksm 3.0.0 oct 19... | 22:42 |
| notmorgan | ok i'm going to flip these then to the latest being unhidden | 22:42 |
| notmorgan | i really hope this doesn't all go explody | 22:42 |
| stevemar | ksm 2.3.2 7 days ago | 22:43 |
| stevemar | crap | 22:43 |
| dhellmann | notmorgan : it'll be fine, pip doesn't pay attention to that flag | 22:43 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Fix indexes in db models https://review.openstack.org/251547 | 22:43 |
| mordred | yah - it's only website | 22:43 |
| anteaya | dhellmann: right, my feeling is that is how the project is set up initially | 22:43 |
| dhellmann | anteaya : right | 22:43 |
| anteaya | dhellmann: by whoever set it up | 22:43 |
| *** hughhalf has joined #openstack-infra | 22:43 | |
| *** btully has joined #openstack-infra | 22:43 | |
| dhellmann | the alternative default is to not hide anything, which means you have a huge list of releases at some point | 22:43 |
| notmorgan | ok pypi is now showing 3.0 as the latest ( stevemar ^ cc dolphm ) | 22:43 |
| stevemar | notmorgan: dhellmann it's cause we did a liberty point release of ksm (marked as 2.3.2) 7 days, so the mitaka release (3.0.0) isn't "newest" since it was 19 days ago | 22:44 |
| notmorgan | dhellmann: yeah i can unset hid old releases auto | 22:44 |
| notmorgan | i'm fine with that | 22:44 |
| *** doug-fis_ has quit IRC | 22:44 | |
| *** mpryc has left #openstack-infra | 22:44 | |
| *** mpryc has quit IRC | 22:44 | |
| stevemar | notmorgan: dhellmann same with keystoneclient btw | 22:44 |
| fungi | mordred: sounds fine | 22:44 |
| *** doug-fish has joined #openstack-infra | 22:44 | |
| *** amitgandhinz has quit IRC | 22:44 | |
| dhellmann | stevemar , notmorgan : yeah, for now a manual change on pypi is the simplest fix | 22:44 |
| anteaya | I think it is the only fix | 22:45 |
| anteaya | that I am aware of | 22:45 |
| stevemar | dhellmann: ty, releases on stable branches should hide, i guess, but not mitaka ? | 22:45 |
| stevemar | notmorgan: can you change keystoneclient too? | 22:45 |
| dhellmann | we could look at making the release tools smarter too but I'm not sure how exactly that works so someone else would have to figure it out | 22:45 |
| notmorgan | ok so i just changed KSM | 22:45 |
| notmorgan | i don't know if i own KSC | 22:45 |
| stevemar | dhellmann: there is something you don't know?! | 22:45 |
| notmorgan | so i might not be able to change it | 22:46 |
| dhellmann | stevemar : many many many things | 22:46 |
| *** dingyichen has joined #openstack-infra | 22:46 | |
| notmorgan | it may need to be an -infra person who has login to pypi access. | 22:46 |
| stevemar | dhellmann: :) | 22:46 |
| SpamapS | https://review.openstack.org/#/c/251536/ <-- this fixes performance issues caused by the recent subunit2sql migration (we forgot to re-add some indexes). | 22:46 |
| notmorgan | i'll fix keystoneauth | 22:46 |
| notmorgan | too | 22:46 |
| stevemar | notmorgan: thanks dude | 22:46 |
| SpamapS | there's some 12-hour long queries running as a result. :-P | 22:46 |
| clarkb | dhellmann it uses twine to push to pypi | 22:46 |
| clarkb | so if twine can set the flag we can do that | 22:47 |
| mordred | SpamapS: yay! | 22:47 |
| fungi | notmorgan: dhellmann: right, this behavior basically surfaced when we started making stable point releases of projects to pypi rather than always rolling forward, so the behavior we were relying on (hide all but the most recently uploaded release) is now sometimes nonsensical | 22:47 |
| stevemar | dolphm: ^ | 22:47 |
| notmorgan | ooh i have keystoneclient i think | 22:47 |
| stevemar | should be mostly fixed... | 22:47 |
| notmorgan | oh no | 22:47 |
| notmorgan | not python-keystoneclient | 22:47 |
| notmorgan | boo | 22:47 |
| notmorgan | yah openstackci needs to update keystoneclient | 22:48 |
| notmorgan | i can't do that one. | 22:48 |
| fungi | we could alternatively decide to not hide any releases, or we could maybe do something smart with our release tooling after upload interfacing with pypi, but that's beyond my ken | 22:48 |
| notmorgan | fungi: i think "not hidin" is the best bet | 22:48 |
| SpamapS | mordred: to be fair .. this was caused by our FK hate. ;) | 22:49 |
| *** doug-fish has quit IRC | 22:49 | |
| anteaya | whatever we decide I think individual project people need to be able to fix it themselves | 22:49 |
| notmorgan | i just flipped keystonemiddleware over to no-hide for everything | 22:49 |
| notmorgan | and keystoneauth | 22:49 |
| anteaya | when they do something they didn't intend on doing | 22:49 |
| anteaya | or they intended and then changed their minds | 22:49 |
| *** doug-fish has joined #openstack-infra | 22:49 | |
| SpamapS | It would be great if you could tell sqlalchemy to run explains on queries during the test suite and fail if they're bad. | 22:50 |
| SpamapS | like just say table scans on test_run_metadata are always fails | 22:50 |
| *** doug-fis_ has joined #openstack-infra | 22:50 | |
| SpamapS | mtreinish: ^ get on that. ;) | 22:50 |
| fungi | "By default, each new release will hide all other release from the regular display. If you want to select yourself which projects to hide, uncheck the button below" | 22:51 |
| mtreinish | SpamapS: there is a --verbose option for all the subunit2sql commands which changes the logging for sqlalchemy to print the sql it's using | 22:51 |
| openstackgerrit | Merged openstack-infra/system-config: Add shade and openstack inventory to system https://review.openstack.org/249929 | 22:51 |
| fungi | so yeah, seems to be a pypi default | 22:51 |
| SpamapS | mtreinish: I like that | 22:51 |
| SpamapS | mtreinish: but I want fail-if-the-db-is-failing-you ;) | 22:52 |
| mordred | ok. new disable rules going in to effect next puppet run | 22:52 |
| mordred | I should maybe send an email to the infra list about that ... | 22:52 |
| mtreinish | SpamapS: ah, I misread what you were asking for :) | 22:52 |
| fungi | notmorgan: so here's how that looks... https://pypi.python.org/pypi/python-keystoneclient | 22:53 |
| mordred | infra-root: the way we disable servers from puppet just changed - the docs have been updated, but it's worth familiarizing yourself with it | 22:53 |
| SpamapS | ianw: when you're around, https://review.openstack.org/#/c/251227/ patchset 7 should work for you | 22:53 |
| fungi | and you need to select a specific version to see the long description/readme and details | 22:53 |
| mordred | infra-root: both methods work at the same time FOR NOW - but that will not be the case for much longer | 22:53 |
| SpamapS | ianw: though it will be very slow until we apply the migration that adds indexes back in. | 22:53 |
| anteaya | that's a lot of versions | 22:54 |
| *** doug-fish has quit IRC | 22:54 | |
| pleia2 | mordred: thanks for the docs updates | 22:54 |
| mtreinish | SpamapS, ianw: jsut tested the script locally and got: http://paste.openstack.org/show/480442/ which of course takes forever because of the missing index :) | 22:55 |
| mordred | pleia2: I figure I should _occasionally_ write docs | 22:55 |
| *** doug-fis_ has quit IRC | 22:55 | |
| pleia2 | :) | 22:55 |
| mtreinish | SpamapS: any issue with fast approving the fixes? | 22:55 |
| jeblair | mordred: "TODO: document" ? ;) | 22:55 |
| mordred | jeblair: too much typing | 22:55 |
| pleia2 | mordred: I've kind of lost track of how we're supposed to do things moving forward, so I really do appreciate it | 22:55 |
| mtreinish | although I guess, sloppy review is what got us in this situation :) | 22:56 |
| mordred | does anyone else experience frequent crappy network connections to puppetmaster? | 22:56 |
| jeblair | mordred: no, and i've spent a lot of time there lately | 22:57 |
| pleia2 | I usually don't, but it's been a little while since I've done a lot of work there | 22:57 |
| nibalizer | mordred: sometimes | 22:57 |
| jeblair | mordred: i'm using v4 | 22:57 |
| nibalizer | my connection to es02 kept dropping today | 22:57 |
| *** mwagner_lap has joined #openstack-infra | 22:58 | |
| fungi | i haven't noticed problem connections to p'master, and i'm always connecting via ipv6 | 22:58 |
| nibalizer | mordred: how can I help with puppet apply? | 22:58 |
| *** yamahata has joined #openstack-infra | 22:58 | |
| jeblair | mordred: sonic-cogent-telia-rax is my route | 22:58 |
| mordred | nibalizer: well, I have now landed the shade/clouds.yaml file with new disable semantics | 22:58 |
| mordred | nibalizer: so next step is verifying that puppet is running normally, and that we've stopped attempting to run it at all on the hosts that are disabled | 22:58 |
| nibalizer | ok | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add more detail to the output https://review.openstack.org/251567 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: catch more errors in external commands https://review.openstack.org/251568 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: better formatting for command errors https://review.openstack.org/251569 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: ensure virtualenv is created in toolsdir https://review.openstack.org/251570 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: factor out command to list deliverable changes https://review.openstack.org/251571 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: rewrite release_from_yaml.py as bash script https://review.openstack.org/251572 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add the series to the release tag https://review.openstack.org/251573 | 22:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email https://review.openstack.org/251574 | 22:59 |
| lifeless | mordred: I've tried to privmsg you, not sure if there is a freenode glitch, or you're just busy :) | 22:59 |
| SpamapS | mtreinish: no I think it's more broken now than it could be with those things. | 22:59 |
| mtreinish | SpamapS: ok, then I'll +A the things :) | 23:00 |
| SpamapS | mtreinish: perhaps we can beg for fast track? | 23:00 |
| *** tlian2 has joined #openstack-infra | 23:00 | |
| mtreinish | SpamapS: well the subunit2sql core group is infra-root, andreaf, masayukig, and oomichi_ | 23:00 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email https://review.openstack.org/251574 | 23:01 |
| nibalizer | mordred: 2015-11-30 23:01:01,029 p=718 u=root | failed: [pypi.regionone.openstack.org -> 127.0.0.1] => MUCH STUFF | 23:01 |
| mordred | looking | 23:01 |
| nibalizer | failing at rsycning system-config | 23:02 |
| nibalizer | this is in puppet_run_all.log | 23:02 |
| *** dims has joined #openstack-infra | 23:02 | |
| SpamapS | mtreinish: I am confident the indexes migration would hurt nothing. The uuid thing I'm pretty sure makes things better, and I like that ianw can confirm that. | 23:02 |
| SpamapS | mtreinish: so I feel good about saying "lets just go" | 23:02 |
| *** doug-fish has joined #openstack-infra | 23:03 | |
| *** dims_ has quit IRC | 23:03 | |
| mtreinish | SpamapS: wfm | 23:03 |
| mordred | nibalizer: .git/.~tmp~ | 23:03 |
| nibalizer | mordred: yea idunno wtf | 23:03 |
| *** tlian has quit IRC | 23:03 | |
| SpamapS | mtreinish: I also don't think we had bad review on the changes up until now. We just didn't have as good test coverage of the public parts of the API as we thought. :) | 23:03 |
| *** changbl has quit IRC | 23:04 | |
| mtreinish | SpamapS: hehe, yeah that's definitely true | 23:04 |
| mtreinish | SpamapS: what about: https://review.openstack.org/#/c/251547/ do we want to fast track that one too? | 23:04 |
| mtreinish | I'm not sure it actually does anything | 23:04 |
| SpamapS | mtreinish: nope | 23:04 |
| *** otter768 has joined #openstack-infra | 23:05 | |
| *** changbl has joined #openstack-infra | 23:05 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Return Run.uuid in public methods https://review.openstack.org/251227 | 23:06 |
| nibalizer | mordred: 2015-11-30 23:04:37,343 p=7042 u=root | failed: [review-dev.openstack.org] => {"disabled": true, "failed": true} | 23:07 |
| *** doug-fish has quit IRC | 23:07 | |
| nibalizer | 2015-11-30 23:04:37,343 p=7042 u=root | msg: Puppet agent is administratively disabled. | 23:07 |
| nibalizer | so i would say our disable is not working | 23:07 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add migration for adding missing indexes https://review.openstack.org/251536 | 23:07 |
| nibalizer | puppet agent disablement is saving us though | 23:07 |
| openstackgerrit | Merged openstack-infra/project-config: This moves the api wg notifications to -sdk https://review.openstack.org/251357 | 23:09 |
| *** otter768 has quit IRC | 23:09 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add the series to the release tag https://review.openstack.org/251573 | 23:10 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: add announce.sh to generate a release notes email https://review.openstack.org/251574 | 23:10 |
| anteaya | fungi pleia2 this would be a new usage of envlist in jenkins/jobs/projects.yaml: https://review.openstack.org/#/c/251190/1/jenkins/jobs/infra.yaml tonyb wants to run two envs on the same node | 23:11 |
| mordred | nibalizer: hahaha | 23:11 |
| anteaya | fungi pleia2 oslo.db does envlist like this: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n3859 | 23:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add a missing colon https://review.openstack.org/251577 | 23:11 |
| mordred | nibalizer: ^^ | 23:11 |
| mordred | nibalizer: that is why the disabled thing did not work for that run | 23:12 |
| anteaya | fungi pleia2 just wondering if you had any objection | 23:12 |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: add seperate job for monasca plugin in devstack install https://review.openstack.org/244893 | 23:12 |
| anteaya | since it is new | 23:12 |
| mordred | nibalizer: I think we need to move our rsync to the puppet role | 23:13 |
| mordred | nibalizer: because I'd argue that if it does not work, we should also not run puppet on that host | 23:13 |
| fungi | anteaya: i think that'll work fine since it'll just call `tox -e ical,check` but you can try that yourself to make sure it's sane | 23:13 |
| nibalizer | mordred: ansible can't just stop once it fails? | 23:13 |
| anteaya | okay thanks | 23:14 |
| mordred | nibalizer: it can - but the sync is in a TOTALLY different playbook | 23:14 |
| SpamapS | ianw: are you still waiting for that query to return? | 23:14 |
| *** sputnik13 has joined #openstack-infra | 23:14 | |
| fungi | anteaya: tonyb: i don't know off the top of my head what the behavior is if ical fails and check succeeds but might be worth confirming it exits nonzero when done | 23:14 |
| mordred | nibalizer: so we'd have to stop ALL puppet operations if rsync fails to any host | 23:14 |
| mordred | nibalizer: which seems - fragile | 23:14 |
| mordred | also - we're doing the rsync for the purposes of running puppet | 23:14 |
| SpamapS | ianw: I ask because the explain suggests it will have to look at 48 billion rows..... | 23:15 |
| ianw | SpamapS: i killed it locally ... hopefully it died on the server | 23:15 |
| SpamapS | ianw: it didn't | 23:16 |
| ianw | hang on, i was just testing right now | 23:16 |
| SpamapS | | 58038 | query | 10.181.6.142:60911 | subunit2sql | Query | 1277 | Sending data | SELECT tests.test_id AS tests_test_id, test_runs.status AS test_runs_status, test_runs.start_time AS | | 23:16 |
| ianw | so i've killed it, as of this second, is it gone now? | 23:16 |
| tsg | lifeless: ping - sorry missed your IRC messages earlier | 23:17 |
| nibalizer | mordred: i keep expecting us to make the ansible-puppet stuff generic enough that someone else could possibly use it, maybe i should get over that | 23:17 |
| ianw | but, even with that latest change from mtreinish, the query still seems to be taking a long time | 23:17 |
| nibalizer | thats my aversion to putting the rsync in place | 23:17 |
| mtreinish | ianw: we need to apply another migration which adds the missing indexes | 23:17 |
| nibalizer | someone else could totally use it... for exactly what we use it for | 23:17 |
| SpamapS | ianw: I see two running, 58038 and 58233 | 23:17 |
| mtreinish | ianw: it should get triggered on the next puppet run on logstash.o.o | 23:17 |
| tsg | lifeless: on liberasurecode pkg availability, we have it on F22, F23, Centos7, precise and trusty at this time | 23:17 |
| *** dtardivel has quit IRC | 23:17 | |
| tsg | lifeless: devstack seems to be ok with those distro revs | 23:18 |
| SpamapS | ianw: yeah, the indexes will build over a few hours :-P | 23:18 |
| lifeless | back soon, picking up family | 23:18 |
| mtreinish | SpamapS: 58038 was probably me, I killed it locally but it still is going on the server side | 23:18 |
| tsg | lifeless: clarkb mentioned that centos6 was going to be unsupported at this point, so hopefull the list above is exhaustive | 23:18 |
| anteaya | fungi: http://paste.openstack.org/show/480445/ | 23:18 |
| SpamapS | mtreinish: kill 58038 took care of it | 23:21 |
| mordred | nibalizer: yah - I think that needs to be a secondary goal | 23:21 |
| SpamapS | lets save all the CPU for index building. :) | 23:21 |
| mordred | nibalizer: although I do agree it's a goal | 23:21 |
| mordred | nibalizer: but - we can make "please rsync my puppet modules" a parameter | 23:21 |
| ianw | SpamapS / mtreinish : ok, well i don't have anything running right now, so anything on the server is a zombie | 23:22 |
| ianw | but, when you release, all the dib builds will start doing that in a thundering herd | 23:22 |
| nibalizer | mordred: ok | 23:23 |
| mordred | nibalizer: patch coming | 23:23 |
| nibalizer | we do need to get a thing working, and validate that it works well before we start building generic parts | 23:23 |
| mtreinish | ianw: heh, yeah it's a race. Will the index builds finish before the nodepool builds get to that script | 23:23 |
| *** peristeri has quit IRC | 23:23 | |
| mtreinish | my money is on the nodepool builds starting first :) | 23:23 |
| SpamapS | ianw: thats good to know. Release happened 10 minutes ago. | 23:24 |
| ianw | 48 billion rows seems like a lot | 23:25 |
| mordred | nibalizer: ++ | 23:25 |
| SpamapS | ianw: right? ;) | 23:26 |
| *** sridhar_ram has quit IRC | 23:26 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-openstack_health: Make sure we install the latest requirements https://review.openstack.org/251564 | 23:27 |
| mtreinish | nibalizer, fungi, mordred: ^^^ that'll be good to land | 23:28 |
| mtreinish | assuming I did it right :) | 23:28 |
| fungi | anteaya: lgtm! | 23:28 |
| fungi | mtreinish: why installing requirements in a separate pass? no pbr? | 23:29 |
| ianw | infra people : since we got DIB back on track ... switching to f23 builds with -> https://review.openstack.org/#/c/247908/ would be great ... | 23:29 |
| anteaya | fungi: thank you | 23:30 |
| *** hichihara has joined #openstack-infra | 23:30 | |
| anteaya | fungi: I appreciate you suggesting how I should test it | 23:30 |
| fungi | anteaya: yw | 23:30 |
| mtreinish | fungi: it has pbr, it's for the -U. Although I guess we could do 'pip install -U -e $repo_path' | 23:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Add puppet module sync to the role https://review.openstack.org/251580 | 23:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use the puppet role to copy the puppet code https://review.openstack.org/251581 | 23:30 |
| mordred | nibalizer: there is the change and the change to consume it | 23:30 |
| fungi | mtreinish: why -e? do you need an editable install there? | 23:30 |
| mtreinish | I didn't really know why it was a seperate thing before, I just was maintaining the status quo | 23:31 |
| mtreinish | fungi: the -e is needed because one of the pbr entry points wasn't getting created without it | 23:31 |
| mtreinish | fungi: http://git.openstack.org/cgit/openstack-infra/puppet-openstack_health/commit/?id=9d625d903d0e3c38fcfe9a88f262db9d6e756e3d | 23:31 |
| mtreinish | fungi: it took nibalizer and I a while to figure that out... | 23:31 |
| fungi | mtreinish: good question. i've not familiarized myself with the nuances of the puppetlabs python module | 23:31 |
| SpamapS | mtreinish: http://puppetboard.openstack.org/node/logstash.openstack.org <-- we should see the migration started here right? | 23:32 |
| fungi | and yeah, lack of pbr entrypoints without an editable install sounds counter to expectations. did you manage to get any input from mordred/lifeless? is it a bug or working as designed? | 23:32 |
| *** ZZelle has quit IRC | 23:33 | |
| mtreinish | fungi: I don't remember, I might have forgotten to follow up on it | 23:33 |
| *** pc_m has left #openstack-infra | 23:33 | |
| mtreinish | I think I bugged mordred about it back in the day though | 23:33 |
| fungi | curious to know the underlying cause there so i can be on the lookout for it in the future | 23:33 |
| nibalizer | mtreinish: seems like maybe we could just add extra_pip_args => '-U' to the requirements resource | 23:33 |
| mordred | if the install was done as editable originally, and then not as editable | 23:33 |
| nibalizer | https://github.com/stankevich/puppet-python/blob/master/manifests/requirements.pp#L38 | 23:33 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make Elastic Recheck Watch more reusable https://review.openstack.org/251515 | 23:34 |
| mordred | I could imagine that something about that sequence would be weird enough | 23:34 |
| mtreinish | SpamapS: yeah it should show up there on the next report | 23:34 |
| mordred | as to be weird | 23:34 |
| fungi | mordred: oh, so maybe deleting the virtualenv on the server and letting it get recreated would have solved it | 23:34 |
| mtreinish | SpamapS: well the new version being picked up will, last week the exec didn't actually show up though | 23:34 |
| jeblair | virtualenv? | 23:34 |
| *** sridhar_ram has joined #openstack-infra | 23:35 | |
| mtreinish | fungi: it was very weird I think I tried it with a fresh venv too and it didn't work then | 23:35 |
| jeblair | why is there a virtualenv? | 23:35 |
| mtreinish | fungi: although it was a while ago | 23:35 |
| *** pc_m has joined #openstack-infra | 23:35 | |
| jeblair | we don't use virtualenvs on servers | 23:35 |
| mtreinish | nibalizer: oh, that might work | 23:35 |
| fungi | yeah, presumably we could just pip install it into the global system environment | 23:35 |
| anteaya | fungi jeblair this patch removes a job from devstack-gate if you care: https://review.openstack.org/#/c/248270/ | 23:36 |
| pc_m | anteaya: hi. need some infra wisdom. | 23:36 |
| anteaya | pc_m: hi | 23:36 |
| anteaya | pc_m: how are you doing? | 23:36 |
| mtreinish | fungi: yeah, I don't know why it's a venv. I didn't write most of the module, just pushed enough to make it work when we first started | 23:36 |
| pc_m | anteaya: stressed out... | 23:36 |
| anteaya | pc_m: :( | 23:36 |
| anteaya | pc_m: I'm sorry to hear that | 23:36 |
| jeblair | mtreinish: this is on logstash.o.o? | 23:37 |
| pc_m | anteaya: need some infra advice. | 23:37 |
| mtreinish | jeblair: health.o.o | 23:37 |
| anteaya | pc_m: let's see if we can find some for you | 23:37 |
| pc_m | Looks like neutron and all the adv svc repos are all failing pep8. | 23:37 |
| mtreinish | jeblair: in reference to: https://review.openstack.org/251564 | 23:37 |
| anteaya | pc_m: can you show me a patch? | 23:37 |
| jeblair | gah | 23:37 |
| nibalizer | jeblair: if thats the policy, it hasn't been well advertised (or followed) | 23:38 |
| fungi | pc_m: i saw that, wasn't sure if it was an epidemic or not. something about netifaces? | 23:38 |
| jeblair | nibalizer: it's been well followed | 23:38 |
| jeblair | except apparently this module | 23:38 |
| pc_m | From what we can gather, it looks like astroid library has updated and it is causing an issue with pylint version we are running. | 23:38 |
| pc_m | pylint 1.4.4 and astroid 1.3.8 work. | 23:38 |
| jeblair | nibalizer, mtreinish: we need to fix that -- it's not cool to have a server that behaves differently from all the others | 23:39 |
| pc_m | Neutron has pylint pinned to 1.5.0 and it brings in astroid 1.4.1 and it causes all sorts of havoc. Many failures. | 23:39 |
| jeblair | and it would be great if when people write new puppet modules they write them to look like our existing ones | 23:39 |
| *** gordc has quit IRC | 23:39 | |
| pc_m | ZZelle was thinking best thing to do is pin astroid at 1.3.8 until we can investigate pylint 1.5.0 and see if it will work with newer astroid. | 23:39 |
| pc_m | However, astroid is not in global requirements, from what I hear. | 23:40 |
| pc_m | How should we proceed? | 23:40 |
| nibalizer | jeblair: we have virtualenvs on puppetboard.o.o and in the askbot stuff too | 23:40 |
| *** xyang1 has quit IRC | 23:41 | |
| *** cbader has joined #openstack-infra | 23:41 | |
| nibalizer | mordred: ansible-lint doesn't like your patch | 23:42 |
| pc_m | Do we need to add astroid to global requirements, and then pin it in neutron and adv svc? | 23:42 |
| cbader | Hi I am having an issue building a dib-image in nodepool. I get an error "mysql_config not found" any clue on what is happening | 23:42 |
| mtreinish | clarkb: did we stop puppet on logstash.o.o again? the last report is > 1hr ago | 23:42 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add rollback script for 2.11 https://review.openstack.org/245598 | 23:42 |
| mordred | nibalizer: that's becuase it doesn't like me | 23:42 |
| pleia2 | mordred: ^^ just cleaned up the whitespace anteaya pointed out so we can get this in | 23:42 |
| pleia2 | (would be lovely to schedule this upgrade this week) | 23:43 |
| jeblair | mtreinish: it looks like folks from thoughtworks wrote that module? | 23:43 |
| mtreinish | jeblair: yeah they did | 23:43 |
| jeblair | nibalizer: i can understand how ask slipped through. puppetboard is :( | 23:44 |
| pc_m | worse with neutron-vpnaas, which doesn't pin pylint either. | 23:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Add puppet module sync to the role https://review.openstack.org/251580 | 23:44 |
| nibalizer | jeblair: well I wrote the puppetboard module before coming here, so thats the history | 23:44 |
| jeblair | mtreinish: can you ask them to update it to stop using virtualenvs so that it matches (nearly all of) the rest of our systems? | 23:44 |
| nibalizer | the puppetboard module is long due of a redo | 23:44 |
| mtreinish | jeblair: I can | 23:44 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:44 | |
| jeblair | nibalizer: ah, so it was probably black-box to us at the time | 23:45 |
| nibalizer | mordred: so you dont want to run install_module.sh on the nodes any more? just sync everything from puppetmater? | 23:45 |
| nibalizer | jeblair: exactly | 23:45 |
| mtreinish | jeblair: well I think I can at least. I don't see anyone on irc right now though :) | 23:45 |
| pc_m | fungi: anteaya: any suggestions? Also, it sounds like pep8-constraints passes. | 23:45 |
| anteaya | pc_m: ah well I don't know exactly how to untangle this | 23:46 |
| fungi | pc_m: so astroid is a dependency of pylint? | 23:46 |
| jeblair | mtreinish: generally we just "pip install -U <thing>" and that's it; should be pretty simple | 23:46 |
| pc_m | fungi: yes and it isn't listed anywhere. Looks like ZZelle said it just updated too. | 23:47 |
| mtreinish | jeblair: sure, that's what I was expecting it to do too | 23:47 |
| fungi | pc_m: yeah, so you would need to add an astroid pin in global requirements if the pylint in global requirements is broken with later astroid. constraints is probably working because it's setting an explicit version of both those and may not have updated (or may be unable to update because of that issue?) | 23:48 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:48 | |
| *** hemna is now known as hemnafk | 23:48 | |
| mtreinish | SpamapS: so it looks like puppet isn't running on logstash.o.o right now. :( | 23:48 |
| SpamapS | mtreinish: doh | 23:49 |
| SpamapS | can we get that turned back on somehow? | 23:49 |
| lifeless | fungi: econtext | 23:49 |
| SpamapS | need those indexes :-P | 23:49 |
| pc_m | fungi: So do we pin to a specific version in global requirements, or just set >=1.3.8 and pin in neutron and others, that are running pylint 1.4.4? | 23:49 |
| jeblair | Nov 30 22:40:22 logstash puppet-agent[6491]: Finished catalog run in 101.81 seconds | 23:50 |
| lifeless | fungi: is the bindep fallback not in use yey? | 23:50 |
| fungi | lifeless: nibalizer and mtreinish have context, but something about openstack-health (which uses pbr) not getting at least one of its pbr entrypoints installed without using editable | 23:50 |
| lifeless | fungi: context: https://review.openstack.org/247936 | 23:50 |
| mtreinish | jeblair: is it just not reporting to puppetboard: http://puppetboard.openstack.org/node/logstash.openstack.org | 23:50 |
| lifeless | fungi: cli scripts I presume? | 23:50 |
| fungi | lifeless: only in an experimental job on nova at the moment. we try to update it when we update the lists in our puppet manifests | 23:50 |
| SpamapS | jeblair: right, over an hour ago. | 23:50 |
| mtreinish | lifeless: wsgi_scripts but yeah | 23:51 |
| lifeless | mtreinish: so wsgi scripts is new | 23:51 |
| nibalizer | lifeless: also this http://git.openstack.org/cgit/openstack-infra/puppet-openstack_health/commit/?id=9d625d903d0e3c38fcfe9a88f262db9d6e756e3d | 23:51 |
| openstackgerrit | Merged openstack-infra/system-config: Invite statusbot to senlin channel https://review.openstack.org/250164 | 23:51 |
| lifeless | mtreinish: its entirely possible its broken, since IIRC it has no tests | 23:51 |
| jeblair | mtreinish, SpamapS: there was a puppet-related ansible log entry from 23:03, but i don't see any actual puppet logs | 23:51 |
| mtreinish | lifeless: well I looked at the code and it didn't seem to be treated any differently than the console_scripts would be | 23:51 |
| jeblair | mtreinish, SpamapS: i think it's time to check in with nibalizer and mordred and see if the apply work is affecting this | 23:52 |
| SpamapS | jeblair: we're expecting it to pick up subunit2sql 1.0.2, much like the worker did. | 23:52 |
| fungi | pc_m: oh, one complication. we don't pin static analysis tools in global requirements/constraints because we let projects specify particular ones, so you won't see pylint there afaik | 23:52 |
| lifeless | fungi: you may wish to weigh in on the pattern in https://review.openstack.org/#/c/247936/ to say its right/wrong | 23:52 |
| mtreinish | lifeless: the issue was when doing just pip install on a repo the wsgi script file wasn't being created. But with install -e it was | 23:52 |
| SpamapS | gah, too many slightly related threads | 23:52 |
| jeblair | SpamapS: good news is that means all of scrollback is relevant! :) | 23:53 |
| fungi | lifeless: just did | 23:53 |
| pc_m | fungi: I pulled requirements repo from github and there is a pylint=1.4.4 | 23:54 |
| pc_m | it is at the end | 23:54 |
| mordred | jeblair: I do not expect there to be any changes that would affect puppet on subunit-worker | 23:54 |
| lifeless | mtreinish: well, I've found one bug from code inspection | 23:54 |
| mtreinish | mordred: this is on logstash.o.o | 23:54 |
| lifeless | mtreinish: wsgi is broken on windows | 23:54 |
| mordred | 2015-11-30 23:03:41,018 p=718 u=root | logstash.openstack.org : ok=4 changed=2 unreachable=0 failed=0 | 23:54 |
| fungi | pc_m: however we skip it in upper-constraints.txt apparently http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt | 23:54 |
| mtreinish | lifeless: haha, well wasn't running it on windows :) | 23:55 |
| mordred | and it's just running agent still | 23:55 |
| lifeless | mtreinish: in local_install_scripts | 23:55 |
| cbader | I found my issue old ubuntu missing files. | 23:55 |
| openstackgerrit | Merged openstack-infra/system-config: Add a missing colon https://review.openstack.org/251577 | 23:55 |
| pc_m | fungi: that's new file to me. Does it imply that the project would have to pin then (and is why neutron does)? | 23:56 |
| jeblair | mordred: how can you tell it's running agent still? | 23:56 |
| jeblair | mordred: or do you mean something other than "there is a currently running puppet agent process"? | 23:57 |
| jeblair | (cause i don't see one) | 23:57 |
| nibalizer | jeblair: ansible runs puppet-agent as of now | 23:57 |
| nibalizer | its safe to just run 'puppet agent --test' on that node if you like | 23:57 |
| pc_m | fungi: upper-constraints has astroid==1.3.8 | 23:57 |
| jeblair | yeah, it's just that it's not running and there was no puppet log output from the 2303 run | 23:57 |
| fungi | pc_m: yep | 23:58 |
| fungi | pc_m: i'm not actually sure about the implication for this as it's getting late in the day for me and things are becoming fuzzy | 23:58 |
| lifeless | right, filed https://bugs.launchpad.net/pbr/+bug/1521407 | 23:58 |
| openstack | Launchpad bug 1521407 in PBR "wsgi entry points not supported on windows" [Wishlist,Triaged] | 23:58 |
| mordred | jeblair: what I mean is that we have not pulled the trigger on anything that would run puppet apply | 23:58 |
| mordred | jeblair: I did not mean "there is an active agent process" | 23:58 |
| mordred | jeblair: sorry for the unclear | 23:58 |
| jeblair | mordred: ok. so before i run --test, i'd like to understand this a bit more... | 23:58 |
| mordred | ++ | 23:59 |
| pc_m | fungi: late for me too. Anyone else from infra I can ask now? | 23:59 |
| openstackgerrit | Merged openstack-infra/project-config: Wrap overlong PROJECTS line in jenkins/jobs/tripleo.yaml https://review.openstack.org/250759 | 23:59 |
| fungi | pc_m: it's more of a requirements/python packaging question | 23:59 |
| jeblair | mordred, nibalizer: our log on pupetmaster says, according to mordred, that it ran puppet agent. the host itself shows no record of that in syslog, only an 'ansible-file' invocation... | 23:59 |
| pc_m | fungi: or should I ask tomorrow and if so, who would e a good person to ask. | 23:59 |
| lifeless | mtreinish: there is a test that it works which calls setup.py install --prefix | 23:59 |
| pc_m | fungi: any idea on who to contact? | 23:59 |
| lifeless | mtreinish: we don't have one testing pip install, but AFAICT things should be equivalent for this purpose | 23:59 |
| mordred | jeblair: oh! wait | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!