| *** pahuang has quit IRC | 00:01 | |
| jeblair | fungi, pabelanger, bkero: aha! a missing piece is that when haveged dumps its 4096 bits into the input pool all at once, that's actually the first addition to the input pool, and so is the *initialization* of the input pool. when a pool is initialized, it's entropy count goes to 0 (/me throws up hands) which causes the spillover routine *not* to be called that time. *subsequent* additions of data to the input pool, however, should ... | 00:01 |
|---|---|---|
| jeblair | ... be eligible for spillover. | 00:01 |
| *** pahuang has joined #openstack-infra | 00:02 | |
| *** spzala has joined #openstack-infra | 00:02 | |
| *** aeng has quit IRC | 00:02 | |
| *** Julien-zte has joined #openstack-infra | 00:02 | |
| fungi | so if it just spammed more than one dump to the seeding input | 00:02 |
| *** zhurong has quit IRC | 00:04 | |
| *** amitgandhinz has quit IRC | 00:04 | |
| *** tqtran has quit IRC | 00:05 | |
| *** sflanigan has quit IRC | 00:05 | |
| jeblair | fungi: yeah, and rngd alone just barely gives it enough entropy to push it over the edge... it might even still require a few seconds worth of interrupts | 00:06 |
| *** tqtran has joined #openstack-infra | 00:06 | |
| jeblair | haveged followed by rngd is probably a recipe for success | 00:07 |
| *** spzala has quit IRC | 00:07 | |
| jeblair | but this is all just serendipity when it works at all | 00:07 |
| *** gomarivera has quit IRC | 00:08 | |
| *** dchen has joined #openstack-infra | 00:08 | |
| *** rfolco has quit IRC | 00:09 | |
| *** Sukhdev has quit IRC | 00:10 | |
| jeblair | fungi: i'm thinking that along bkero's line of thinking -- we write a systemd unit file that runs haveged to get say 4kbits of data or however much is needed to: 1) add 128 bits to the input buffer so it's initialized, then fill enough of the input buffer that the spillover happens and nonblocking is initialized | 00:11 |
| *** Sukhdev has joined #openstack-infra | 00:12 | |
| jeblair | actually, the easiest/most robust way to do this might be to open up urandom, read a byte from it in one thread, and as long as that thread is blocked, add data from haveged using the ioctl. | 00:12 |
| jeblair | (and run this all from a very early systemd unit) | 00:13 |
| jeblair | pabelanger, mordred, bkero: ^ | 00:14 |
| *** kzaitsev_mb has quit IRC | 00:14 | |
| *** sarob has quit IRC | 00:15 | |
| *** aeng has joined #openstack-infra | 00:16 | |
| *** gomarivera has joined #openstack-infra | 00:17 | |
| *** sflanigan has joined #openstack-infra | 00:18 | |
| *** sflanigan has joined #openstack-infra | 00:18 | |
| *** mtanino has quit IRC | 00:18 | |
| *** cody-somerville has quit IRC | 00:19 | |
| *** shashank_hegde has quit IRC | 00:20 | |
| *** _ari_ has quit IRC | 00:20 | |
| *** Hal has quit IRC | 00:21 | |
| *** vinaypotluri has quit IRC | 00:21 | |
| *** hockeynut has joined #openstack-infra | 00:22 | |
| fungi | yeah, that gets us a signal when it's ready basically | 00:23 |
| *** Julien-zte has quit IRC | 00:23 | |
| fungi | and then we can block anything else that needs urandom given sufficient dependency declaration | 00:23 |
| *** thorst has joined #openstack-infra | 00:24 | |
| *** Julien-zte has joined #openstack-infra | 00:25 | |
| *** Julien-zte has quit IRC | 00:26 | |
| *** Apsu has left #openstack-infra | 00:28 | |
| *** thorst has quit IRC | 00:31 | |
| *** gomarivera has quit IRC | 00:32 | |
| *** pvaneck has quit IRC | 00:32 | |
| *** gomarivera has joined #openstack-infra | 00:33 | |
| *** mriedem has joined #openstack-infra | 00:35 | |
| *** gomarivera has quit IRC | 00:37 | |
| *** adrian_otto has joined #openstack-infra | 00:39 | |
| *** gildub has joined #openstack-infra | 00:39 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects https://review.openstack.org/347047 | 00:41 |
| fungi | reading https://lwn.net/Articles/686033/ it looks like the coming linux rng is switching to the same csprng openbsd's been using for years | 00:42 |
| *** adrian_otto has quit IRC | 00:43 | |
| jeblair | fungi: i don't think we even need to worry about deps -- the fact that urandom itself is blocking will take care of that | 00:44 |
| *** thorst has joined #openstack-infra | 00:45 | |
| *** gyee has quit IRC | 00:46 | |
| jeblair | mtreinish: self.labels on the NodeWorker should have what you're looking for -- http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n568 | 00:46 |
| fungi | jeblair: oh, true, the main problem was unbound, and it was waiting to be able to read pseudorandom bits anyway | 00:48 |
| jeblair | mtreinish: it's actually a list, though in practice, it's going to be a single element for us. though i think that we should still pass it through zmq json as a list, then use the first element of the list in the logstash worker | 00:49 |
| bkero | Just need to make sure the new rng unit service runs and doesnt block on unbound | 00:49 |
| *** Vmem-intern has joined #openstack-infra | 00:49 | |
| *** ihrachys has quit IRC | 00:50 | |
| jeblair | bkero: yep; ideally it might be nice to run before haveged too. | 00:50 |
| Vmem-intern | Hello everyone, would anyone be available to answer a question about a CI error I keep getting on all tests | 00:50 |
| *** ihrachys has joined #openstack-infra | 00:51 | |
| *** tqtran has quit IRC | 00:51 | |
| anteaya | Vmem-intern: it is best if you just ask your question, sharing any logs in paste.openstack.org, as well as stating what steps you have taken to find your own answer | 00:52 |
| anteaya | Vmem-intern: if someone has the time and inclination to help you, they will | 00:52 |
| Vmem-intern | sounds great thank you | 00:54 |
| anteaya | you're welcome | 00:54 |
| *** chlong has joined #openstack-infra | 00:58 | |
| *** amitgandhinz has joined #openstack-infra | 01:00 | |
| *** hockeynut has quit IRC | 01:02 | |
| *** hockeynut has joined #openstack-infra | 01:02 | |
| *** rfolco has joined #openstack-infra | 01:03 | |
| *** rfolco has quit IRC | 01:03 | |
| *** jimbaker has quit IRC | 01:04 | |
| *** spzala has joined #openstack-infra | 01:04 | |
| *** ihrachys has quit IRC | 01:04 | |
| anteaya | 36 in the gate and the little one said, roll over, roll over | 01:05 |
| *** raunak has joined #openstack-infra | 01:05 | |
| anteaya | so they all rolled over and one fell out | 01:05 |
| *** amitgandhinz has quit IRC | 01:05 | |
| *** jimbaker has joined #openstack-infra | 01:07 | |
| *** jimbaker has quit IRC | 01:07 | |
| *** jimbaker has joined #openstack-infra | 01:07 | |
| *** dimtruck is now known as zz_dimtruck | 01:08 | |
| *** zhurong has joined #openstack-infra | 01:09 | |
| *** woodster_ has quit IRC | 01:09 | |
| anteaya | it appears that ci-backup-rs-ord is not reporting, and hasn't been for a while now: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2984&rra_id=all | 01:09 |
| *** spzala has quit IRC | 01:09 | |
| anteaya | also we might want to re-evaluate etherpad storage before summit: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=120&rra_id=all | 01:11 |
| *** baoli has joined #openstack-infra | 01:11 | |
| anteaya | should firehose be reporting anything yet? http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=300 | 01:12 |
| *** shashank_hegde has joined #openstack-infra | 01:13 | |
| *** Apoorva has quit IRC | 01:13 | |
| *** Vmem-intern has quit IRC | 01:13 | |
| anteaya | do we still need jenkins - 07 in cacti? | 01:14 |
| *** sarob has joined #openstack-infra | 01:14 | |
| asselin | I have some questions about masterless puppet used by -infra. ansible_hostfile here is a directory: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/puppetmaster.pp#n18 , shouldn't it be the file in line 23? | 01:15 |
| anteaya | jenkins has been removed in http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/common.yaml#n224 but is still showing up in the navigation in cacti | 01:15 |
| asselin | so I get an error when I try to do the same thing with expand_groups http://paste.openstack.org/show/558617/ | 01:16 |
| *** Sukhdev has quit IRC | 01:16 | |
| anteaya | I don't know what kdc 01 and 02 are but 02 has some interesting traffic patterns: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2291&rra_id=all | 01:17 |
| anteaya | and if we are open to renaming them I vote for xkdc | 01:17 |
| ianw | how do people think that TWO separate reviewers are going to spend their time effectively reverse-engineering their change when they can't be bothered writing a change description | 01:18 |
| ianw | </rant> | 01:18 |
| *** rbuzatu has quit IRC | 01:19 | |
| anteaya | I concur a good commit message really helps a patch move along | 01:20 |
| anteaya | pabelanger: why does pbx take so much space? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=798&rra_id=all | 01:20 |
| cloudnull | evenings | 01:20 |
| *** Apoorva has joined #openstack-infra | 01:21 | |
| anteaya | evenings | 01:22 |
| anteaya | fungi: cacti tells me the wiki seems to be not online: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=355&rra_id=all | 01:22 |
| anteaya | https://wiki.openstack.org/wiki/ gets me can't access the database | 01:22 |
| anteaya | fungi: http://imgur.com/a/YWcDv | 01:24 |
| jeblair | 24464 elastic+ 20 0 251340 10368 0 S 389.8 0.1 231:38.08 kill | 01:24 |
| anteaya | seems to have gone offline about an hour ago | 01:24 |
| jeblair | 1933 elastic+ 20 0 3510620 1.433g 27360 S 0.7 18.3 43:42.31 java | 01:25 |
| *** shashank_hegde has quit IRC | 01:26 | |
| fungi | anteaya: yikes, that's no good. looking into it now | 01:26 |
| anteaya | fungi: thanks | 01:26 |
| jeblair | fungi: ^ | 01:26 |
| jeblair | kill 1018 elasticsearch txt REG 202,1 751360 360459 /tmp/kill (deleted) | 01:26 |
| fungi | elasticsearch eating a ton of ram | 01:26 |
| fungi | yeah | 01:26 |
| fungi | wiki page just loaded for me | 01:27 |
| anteaya | oh good | 01:27 |
| *** shashank_hegde has joined #openstack-infra | 01:27 | |
| anteaya | fungi: what do you make of this page? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=355&rra_id=all | 01:27 |
| fungi | i'm guessing something kicks off to reindex the search data around midnight utc? | 01:27 |
| jeblair | and it's called "/tmp/kill" ? | 01:28 |
| fungi | it's taking a while to ssh in | 01:28 |
| jeblair | also, we run an elasticsearch on wiki now? | 01:30 |
| fungi | have since forever. it's the backend for the wiki search extension | 01:30 |
| jeblair | neat | 01:30 |
| fungi | but yeah, this looks odd | 01:31 |
| fungi | 32 of them in the process table | 01:31 |
| *** shashank_hegde has quit IRC | 01:32 | |
| jeblair | just going to throw out an idea... | 01:32 |
| fungi | also a /tmp/4.4 and a /dev/shm/kill | 01:32 |
| fungi | this looks pretty pwnd | 01:33 |
| jeblair | there is no firewall on wiki.openstack.org. elasticsearch is notoriously insecure and expects to be protected. perhaps the server has been compromised. | 01:33 |
| anteaya | jeblair: I think we need to just assume that is true | 01:33 |
| anteaya | as we really have no way to prove otherwise, do we? | 01:33 |
| fungi | no iptables rules loaded? | 01:33 |
| *** adrian_otto has joined #openstack-infra | 01:34 | |
| fungi | and /dev/shm/kill does exist. probably a rootkit | 01:34 |
| fungi | yeah, iptables rules loaded here are empty | 01:35 |
| anteaya | does the old wiki server still exist? | 01:35 |
| *** esberglu has joined #openstack-infra | 01:35 | |
| anteaya | so we would lose 2 working days of wikipages, if we moved back to the old server, yes? | 01:36 |
| *** nwkarsten has joined #openstack-infra | 01:36 | |
| fungi | sorry, was about to go to sleep. needed to switch back to my office | 01:37 |
| fungi | yeah, i made a snapshot at the time of the upgrade | 01:37 |
| cloudnull | fungi: any progress with that image? | 01:37 |
| anteaya | fungi: yup, I understand | 01:37 |
| fungi | cloudnull: nope, new emergencies | 01:37 |
| fungi | dropped like a hot potato | 01:37 |
| cloudnull | oh. ok. no worries. :) | 01:37 |
| anteaya | fungi: do we want to create a workflow on an etherpad? | 01:38 |
| fungi | i might get back to it later in the week, but i have no actual evidence that our traceroute6 in osic is in any way indicative of other problems | 01:38 |
| anteaya | or use the one you had for the move to the current server? | 01:38 |
| *** adrian_otto has quit IRC | 01:39 | |
| *** cody-somerville has joined #openstack-infra | 01:40 | |
| jeblair | telnet wiki.openstack.org 9200 | 01:40 |
| jeblair | that works, btw | 01:40 |
| jeblair | there is no /etc/iptables/rules file | 01:40 |
| cloudnull | fungi: I also cant reproduce it, and i've built at least one instance on every compute node to test... I'll keep poking around to see if i can find something else. | 01:40 |
| fungi | yeah, this is probably my fault if the iptables-persistent package upgrade from precise to trusty moved the /etc/iptables/rules file or something | 01:41 |
| fungi | and i didn't notice, and then i rebooted the server | 01:41 |
| fungi | and boom, no v4 packet filtering | 01:41 |
| fungi | we should probably go ahead and temporarily shut the server down | 01:42 |
| jeblair | fungi: puppet would have put it back | 01:42 |
| fungi | jeblair: i had to disable puppet before the upgrade because our mediawiki puppet module isn't suitable for trusty | 01:42 |
| fungi | nor for a bunch of the experimental configuration of jpmaxman's i ported over from the upgrade test server | 01:43 |
| anteaya | fungi: so no wiki tonight? or put your snapshot on a different server? | 01:43 |
| jeblair | ya, this may be a good reason to do the next go-around with puppet | 01:43 |
| *** yanyanhu has joined #openstack-infra | 01:44 | |
| fungi | anteaya: i'm going to try booting the snapshot i saved. the database is on trove, so probably reusable (also we have snapshots) | 01:44 |
| anteaya | jeblair: I am firmly in that camp | 01:44 |
| anteaya | I also have moved my tent to the robots.txt no index camp as well | 01:44 |
| anteaya | fungi: cool, let me know what if anything I can do to help | 01:45 |
| cloudnull | interestingly i can however reproduce this issue when I use the infra image... :\ | 01:45 |
| jeblair | fungi: we may also have server backups, i'm checking | 01:45 |
| *** julim has joined #openstack-infra | 01:46 | |
| * craige needs to create a startup file for a daemon and wonders which method is preferred for infra. | 01:47 | |
| cloudnull | pabelanger: didn't the infra image update go into place w/ the v6 changes for unbount ? | 01:47 |
| cloudnull | *unbound | 01:47 |
| jeblair | fungi: perhaps we should move to incident? | 01:47 |
| fungi | just did | 01:47 |
| *** zz_dimtruck is now known as dimtruck | 01:49 | |
| *** esberglu has quit IRC | 01:49 | |
| *** sarob has quit IRC | 01:49 | |
| *** fguillot has quit IRC | 01:52 | |
| *** tqtran has joined #openstack-infra | 01:53 | |
| *** raunak has quit IRC | 01:55 | |
| *** tqtran has quit IRC | 01:57 | |
| cloudnull | fungi: I know you're off fighting fires, but i think i've found the issue with the image. I noticed that the infra image is using the net.ipv6.conf.*.use_tempaddr sysctl which is causing the instance to have multiple IPv6 interfaces. If I disable that traceroute and other bits all work. | 02:01 |
| cloudnull | infra images have net.ipv6.conf.all.use_tempaddr=2 & net.ipv6.conf.default.use_tempaddr=2 I set those to net.ipv6.conf.all.use_tempaddr=0 & net.ipv6.conf.default.use_tempaddr=0 and all was good to go | 02:02 |
| *** amitgandhinz has joined #openstack-infra | 02:02 | |
| cloudnull | net.ipv6.conf.*.use_tempaddr=0 is the default within the ubuntu cloudimage | 02:02 |
| *** aeng has quit IRC | 02:04 | |
| jpmaxman | ouch. fungi the DB is going to be for 1.27 | 02:05 |
| cloudnull | i'm not sure where I'd make such a change to effect the infra images, but it seems like disabling the privacy extension for cloud images would be a worth while to update improve things especially with private addresses having preference. | 02:05 |
| *** gongysh has joined #openstack-infra | 02:05 | |
| *** thorst has quit IRC | 02:06 | |
| *** thorst has joined #openstack-infra | 02:06 | |
| *** amitgandhinz has quit IRC | 02:07 | |
| jpmaxman | fungi: I could probably get a new trusty server up and running pretty fast. I assumed the firewall rules would persist given it was a clone, but I guess you're saying the in place trusty upgrade wiped them? | 02:07 |
| fungi | jpmaxman: yeah, i think the upgrade from precise to trusty moved the iptables rules out of the way, reboot brought it up with no filtering, and exposed the elasticsearch api which is notoriously insecure. stir in script kiddies and you have rootkit soup | 02:07 |
| fungi | jpmaxman: we're currently evaluating whether to point dns at wiki-upgrade-test.o.o with the production trove db, but i also have the snapshot i created on friday booting up for another option | 02:08 |
| fungi | in theory filesystem contents should be stable since we disabled uploads weeks ago | 02:09 |
| *** gongysh has quit IRC | 02:10 | |
| *** asettle has joined #openstack-infra | 02:10 | |
| *** baoli has quit IRC | 02:10 | |
| cloudnull | ouch... ^ if any help is needed i'd be happy to lend a hand. | 02:11 |
| jpmaxman | fungi: ok let me know if I can do anything... you are right that the filesystem should be static. So if you have a pre-compromised image that would probably do the trick. | 02:11 |
| *** baoli has joined #openstack-infra | 02:11 | |
| * cloudnull obviously not on the infra team but still willing to help where i cna. | 02:11 | |
| *** sflanigan has quit IRC | 02:12 | |
| jpmaxman | fungi: also a fresh 1.27 install on a fresh trusty machine would be pretty easy to accomplish. Would just need to grab the skin and a few extensions from the existing server. | 02:12 |
| *** gomarivera has joined #openstack-infra | 02:12 | |
| openstackgerrit | guo yunxian proposed openstack/os-testr: Update home-page url https://review.openstack.org/356179 | 02:12 |
| *** sdake has quit IRC | 02:13 | |
| fungi | jpmaxman: yep, i plan to be doing that tomorrow anyway as i test the wiki-dev build with Krenair's puppet changes | 02:13 |
| anteaya | cloudnull: thank you | 02:13 |
| *** sdake has joined #openstack-infra | 02:13 | |
| jpmaxman | fungi: ok well I'm here at least for the next hour or so if you need anythign don't hesitate | 02:13 |
| anteaya | cloudnull: the main requirement for being on the team is willingness to help | 02:14 |
| anteaya | cloudnull: other skills appreciated but not required | 02:14 |
| cloudnull | :) | 02:14 |
| anteaya | :) | 02:14 |
| *** thorst has quit IRC | 02:15 | |
| *** julim has quit IRC | 02:17 | |
| *** asettle has quit IRC | 02:17 | |
| *** aeng has joined #openstack-infra | 02:18 | |
| *** elo has quit IRC | 02:19 | |
| *** zhurong_ has joined #openstack-infra | 02:19 | |
| *** rbuzatu has joined #openstack-infra | 02:20 | |
| *** yamahata has quit IRC | 02:20 | |
| *** tphummel has quit IRC | 02:21 | |
| *** zhurong has quit IRC | 02:23 | |
| *** rbuzatu has quit IRC | 02:25 | |
| *** sflanigan has joined #openstack-infra | 02:25 | |
| *** gomarivera has quit IRC | 02:25 | |
| *** gomarivera has joined #openstack-infra | 02:25 | |
| *** markusry has quit IRC | 02:25 | |
| *** adrian_otto has joined #openstack-infra | 02:27 | |
| *** spzala has joined #openstack-infra | 02:29 | |
| *** spzala has quit IRC | 02:30 | |
| *** spzala has joined #openstack-infra | 02:30 | |
| fungi | https://wiki.openstack.org/ is now being served from the wiki-upgrade-test.openstack.org snapshot | 02:31 |
| fungi | using our production db still in the same trove instance | 02:31 |
| fungi | which, thankfully, still had firewall rules due to puppet not getting disabled until after it was upgraded to trusty | 02:32 |
| *** adrian_otto has quit IRC | 02:33 | |
| fungi | also the compromised wiki.o.o has been shut down for the time being | 02:33 |
| anteaya | yay | 02:33 |
| anteaya | thank you fungi | 02:33 |
| cloudnull | nice! | 02:33 |
| anteaya | did we lose any wiki edits do you think? | 02:34 |
| anteaya | should be post to the mailing list? | 02:34 |
| *** gongysh has joined #openstack-infra | 02:35 | |
| fungi | i don't think we'd have lost any. i shut down the old wiki before i brought the replacement back into service | 02:35 |
| anteaya | okay | 02:35 |
| anteaya | thank you | 02:35 |
| fungi | there weren't two copies of the db in play, so no chance for edits to be made through the old server not reflected by the replacement | 02:36 |
| jpmaxman | fungi: good show - I'm a bit confused about the order of events but I'm just glad you got i t back up so quickly | 02:36 |
| jpmaxman | glad we moved to trove :) | 02:36 |
| fungi | yep, that saved me having to try to mysqldump and copy | 02:36 |
| fungi | #status log DNS for wiki.openstack.org currently goes to the wiki-upgrade-test.openstack.org server, as the former suffered a compromise due to missing iptables rules | 02:37 |
| *** Apoorva_ has joined #openstack-infra | 02:37 | |
| openstackstatus | fungi: finished logging | 02:37 |
| anteaya | ah I see | 02:37 |
| fungi | jpmaxman: order of events, i reckon, is that when i upgraded wiki.o.o from precise to trusty on friday the iptables-persistent package tried to move our /etc/iptables/rules file to /etc/iptables/rules.v4 | 02:38 |
| fungi | however, we already had /etc/iptables/rules.v4 as a symlink to /etc/iptables/rules (as a backward-compat configuration to make that work on both precise and trusty) | 02:39 |
| *** zhurong_ has quit IRC | 02:39 | |
| fungi | so we ended up with /etc/iptables/rules.v4 as a dangling symlink to a nonexistent /etc/iptables/rules file | 02:39 |
| *** baoli has quit IRC | 02:39 | |
| * amrith perks up when he hears "glad we moved to trove" and hopes it was not sarcastic :) | 02:39 | |
| fungi | amrith: totally non-sarcastic | 02:39 |
| *** zhurong has joined #openstack-infra | 02:39 | |
| amrith | hey, another happy customer!!!! | 02:40 |
| jpmaxman | fungi: for some reason I thought the original move was a DNS repoint to wiki-upgrade-test server... didn't realize you did an in place upgrade of wiki.o.o | 02:40 |
| amrith | made my day | 02:40 |
| Krenair | is the proper iptables rule in puppet? | 02:40 |
| jpmaxman | fungi: so why would wiki-upgrade-test have held onto the iptables? | 02:40 |
| fungi | jpmaxman: i didn't notice that, and then rebooted after the trusty upgrade for the newer linux kernel that came wit the upgrade, resulting in iptables-persistent loading an empty ipv4 filter ruleset, so wide-open | 02:40 |
| *** thorst has joined #openstack-infra | 02:40 | |
| *** thorst has quit IRC | 02:40 | |
| *** Apoorva has quit IRC | 02:40 | |
| fungi | jpmaxman: in the case of wiki-upgrade-test i forgot to disable puppet until after you had upgraded it | 02:41 |
| *** thorst has joined #openstack-infra | 02:41 | |
| *** Apoorva_ has quit IRC | 02:41 | |
| jpmaxman | I didn't think pupppet was ever running against wiki-upgrade-test | 02:41 |
| fungi | if you'll recall, puppet changed the firewall rules on it to only allow ssh access (because we don't have a specific profile for wiki-upgrade-test so it didn't know we'd want things like http/https open) | 02:41 |
| *** gomarivera has quit IRC | 02:41 | |
| fungi | we run puppet against all our servers unless we explicitly disable them | 02:42 |
| fungi | ansible's openstack inventory gets pointed at an openstack tenant and it builds a hostlist from all servers returned by the nova api | 02:42 |
| jpmaxman | gotcha | 02:42 |
| fungi | and we apply puppet to all of them. but if we don't have a specific entry for them to get special additional configuration all it does is lock down iptables to only allow ssh, and add our user accounts | 02:43 |
| fungi | and install a handful of common utilities we want on all servers | 02:43 |
| fungi | basically anything in or referenced from our generic openstack_project::server class | 02:43 |
| *** spzala has quit IRC | 02:44 | |
| jpmaxman | cool - feelin the magic ;) Alright, well I'm glad in this case at least it sort of worked out. Onward and upward to a proper puppetized wiki | 02:44 |
| *** spzala has joined #openstack-infra | 02:44 | |
| fungi | so anyway, by the time i discovered that puppet was still configuring wiki-upgrade-test.o.o and disabled that, it would have already re-fixed the iptables rules which could have been broken by the upgrade to trusty | 02:44 |
| fungi | whereas for the wiki.o.o in-place upgrade, i remembered that i needed to disable puppet before upgrading or else it would undo the extension adjustments in Settings.php, apache 2.4 changes in the vhost config, et cetera | 02:45 |
| jpmaxman | ok well thanks for jumpin on it fungi | 02:46 |
| fungi | Krenair: yes, the proper iptables rules for it are absolutely in puppet: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/wiki.pp#n22 | 02:48 |
| *** spzala has quit IRC | 02:48 | |
| jpmaxman | let me know when we have the dev wiki up and I'll be happy to take a look | 02:49 |
| fungi | i'm shooting to start trying to launch it with Krenair's puppet patches tomorrow | 02:49 |
| *** thorst has quit IRC | 02:49 | |
| jpmaxman | cool | 02:50 |
| fungi | and maybe we'll have this properly managed through revision control and configuration management by the weekend | 02:50 |
| *** yuanying has quit IRC | 02:50 | |
| *** elo has joined #openstack-infra | 02:50 | |
| fungi | depending on what other emergencies land in my lap | 02:50 |
| *** csomerville has joined #openstack-infra | 02:50 | |
| *** raunak has joined #openstack-infra | 02:51 | |
| anteaya | you idealist you | 02:52 |
| fungi | the boot from my pre-upgrade snapshot did eventually boot. i have it up for the time being in case we need it as a reference (as opposed to leaving the compromised one still online) | 02:53 |
| *** sarob has joined #openstack-infra | 02:53 | |
| fungi | though it's not using trove, still local mysql db | 02:53 |
| anteaya | thank you | 02:53 |
| anteaya | how long did it take to boot? | 02:53 |
| *** cody-somerville has quit IRC | 02:54 | |
| fungi | best guess somewhere around 30 minutes | 02:54 |
| fungi | also i won't put it in dns for now, but it's 2001:4800:7813:516:be76:4eff:fe04:802d/192.237.202.184 | 02:54 |
| anteaya | thanks, was just wondering in case something happens and someone needs to access the server | 02:54 |
| anteaya | I've had enough fun for one day | 02:54 |
| anteaya | g'night | 02:54 |
| fungi | the old hackwiki is 2001:4800:7810:512:3bc3:d7f6:ff04:5371/166.78.19.73 but in a powered off state | 02:54 |
| anteaya | thanks for being around | 02:54 |
| fungi | anteaya: thank you for spotting that the server was behaving questionably! | 02:55 |
| pabelanger | anteaya: A quick looks show it is log files that are filling up pbx.o.o | 02:59 |
| ianw | i bet the unix timestamp will be > 2001:4800:7810:512:3bc3:d7f6:ff04:5371 when my isp finally figures out ipv6 | 02:59 |
| pabelanger | I also crashed asterisk, which people were using :( So that will be the reason if people are complaining | 03:00 |
| fungi | ianw: same here. https://tunnelbroker.net/ ftw | 03:00 |
| pabelanger | it is restarted again, but calls were dropped | 03:00 |
| fungi | ianw: my openbsd firewall at home has a gif0 interface that routes my global ipv6 traffic to/from hurricane electric, so from my lan everything looks like normal dual-stack goodness | 03:01 |
| ianw | fungi: yeah, i did that, setup my router and all. the problem is that the only people who have figured out ipv6 are people like google/youtube/facebook/netflix, who are also the people who peer with isps to get content close to you. so you kind of shoot yourself in the foot latency & throughput wise on everything that matters | 03:02 |
| fungi | ianw: fair enough | 03:02 |
| fungi | i agree that's a kinda painful upshot | 03:02 |
| *** amitgandhinz has joined #openstack-infra | 03:03 | |
| ianw | yeah, my "clients" (i.e. wife & children) complained to management and i had to revert :) | 03:03 |
| fungi | ianw: you need your own vlan ;) | 03:03 |
| fungi | with a wap that can put different ssids on different 802.1q vlans, you can make their personal network v4-only and your wfh network dual-stack | 03:05 |
| fungi | i'm doing something similar for my guest wireless | 03:05 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/devstack-gate: Disabled IPv6 privacy extensions https://review.openstack.org/356196 | 03:06 |
| fungi | but yes, would be so much simpler all around if residential isps figured out how to ipv6 | 03:06 |
| cloudnull | IDK if thats where we'd perfer those types of things , but there it is . | 03:06 |
| cloudnull | fungi: ++ timewarner had v6 then took it away... | 03:06 |
| fungi | cloudnull: seriously? that makes me glad i moved out of raleigh | 03:07 |
| fungi | not that being on charter is any better | 03:07 |
| *** jerryz has joined #openstack-infra | 03:07 | |
| cloudnull | seriously. | 03:07 |
| cloudnull | im in San Antonio and was happy w/ my v6 access then, nothing. | 03:07 |
| fungi | was it like a limited beta and then they decided to redo everything, or what? | 03:07 |
| *** amitgandhinz has quit IRC | 03:07 | |
| cloudnull | they hit an issue with the modems they we're using. | 03:08 |
| cloudnull | and instead of replacing them, they pulled v6 | 03:08 |
| ianw | cloudnull: wouldn't that be more appropriate in node setup scripts? what about non d-g jobs? | 03:08 |
| fungi | oh, right, i remember not long before i moved twc switched from motorola to ubee or something | 03:08 |
| cloudnull | ianw: maybe? | 03:09 |
| cloudnull | where are those scripts. | 03:09 |
| cloudnull | sorry for my ignorance. | 03:09 |
| fungi | ianw: cloudnull: i would actually bake those sysctl.conf additions into our image builds | 03:09 |
| *** sarob has quit IRC | 03:10 | |
| fungi | though i guess that makes the detection method in 356196 problematic | 03:10 |
| fungi | oh, no it doesn't | 03:10 |
| *** mriedem has quit IRC | 03:11 | |
| fungi | skimming i thought you were querying sysctl output for current state but you're grepping sysctl.conf to decide as well | 03:11 |
| fungi | so should be able to just put that in one of our nodepool dib elements i think | 03:11 |
| cloudnull | yea, i figured if its not there, set-it, if it is there, set-it to what we want it to be regardless of what it is . | 03:12 |
| fungi | thinking the nodepool-base element at http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base would be a good place | 03:12 |
| *** devkulkarni has joined #openstack-infra | 03:13 | |
| ianw | cloudnull fungi: ++ on putting it there to catch all nodes | 03:13 |
| fungi | just as a several-line script in the install.d tree there | 03:13 |
| cloudnull | on it now. | 03:14 |
| fungi | i'm 100% in favor of dropping v6 privacy extensions on our workers | 03:14 |
| fungi | and on our persistent servers too, but i'd probably do that separately in our openstack_project::server puppet class or something | 03:15 |
| fungi | okay, i'm heading back to bed before it gets too much later | 03:18 |
| *** r-mibu has quit IRC | 03:19 | |
| ianw | bye! | 03:19 |
| *** adrian_otto has joined #openstack-infra | 03:23 | |
| *** kushal has quit IRC | 03:24 | |
| *** r-mibu has joined #openstack-infra | 03:25 | |
| *** bswartz has joined #openstack-infra | 03:26 | |
| *** nwkarsten has quit IRC | 03:27 | |
| *** nwkarsten has joined #openstack-infra | 03:28 | |
| *** adrian_otto has quit IRC | 03:28 | |
| *** aeng has quit IRC | 03:29 | |
| *** vikrant has joined #openstack-infra | 03:30 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Disabled IPv6 privacy extensions https://review.openstack.org/356204 | 03:31 |
| cloudnull | ianw fungi: ^ i think that should do it? | 03:31 |
| *** nwkarsten has quit IRC | 03:32 | |
| ianw | cloudnull: hmm, we have /sys mounted from the build system into the chroot environment we're building at that point | 03:34 |
| *** sdake has quit IRC | 03:34 | |
| *** sflanigan has quit IRC | 03:34 | |
| ianw | so basically it doesn't correspond to the runtime system | 03:35 |
| cloudnull | so maybe i dont try and detect the value and simply write to /etc/sysctl.conf ? | 03:35 |
| *** zhurong has quit IRC | 03:35 | |
| *** tphummel has joined #openstack-infra | 03:35 | |
| *** zhurong has joined #openstack-infra | 03:36 | |
| *** gomarivera has joined #openstack-infra | 03:37 | |
| *** jerryz has quit IRC | 03:37 | |
| ianw | cloudnull: i think, and i'm just reading up on it now, that just setting "net.ipv6.conf.all.use_tempaddr = 0" unconditionally in there will do what we want? | 03:38 |
| cloudnull | all and default | 03:38 |
| ianw | cloudnull: right ... because you can't assume what the interfaces will be on the booted system, or their name | 03:39 |
| *** baoli has joined #openstack-infra | 03:41 | |
| *** oanson has joined #openstack-infra | 03:41 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Disabled IPv6 privacy extensions https://review.openstack.org/356204 | 03:43 |
| cloudnull | ok so that should just look for the setting and do the needful | 03:43 |
| *** devkulkarni1 has joined #openstack-infra | 03:44 | |
| *** devkulkarni has quit IRC | 03:44 | |
| *** baoli has quit IRC | 03:45 | |
| ianw | cloudnull: i don't think it wants to look for the setting, we should be right to just put it into sysctl.conf | 03:46 |
| ianw | because it's just going to be seeing the settings of the build system | 03:46 |
| *** aeng has joined #openstack-infra | 03:46 | |
| cloudnull | that executing out the chroot right? | 03:47 |
| *** thorst has joined #openstack-infra | 03:48 | |
| cloudnull | assuming that it is, that should be hitting /etc/sysctl.conf within the chroot and just looking to see if the settings are already present. | 03:48 |
| cloudnull | no longer actually scrapping sysctl or writing | 03:48 |
| cloudnull | though i may still be confused there. | 03:48 |
| * cloudnull not super familiar with DIB | 03:48 | |
| *** yuanying has joined #openstack-infra | 03:49 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Disabled IPv6 privacy extensions https://review.openstack.org/356204 | 03:50 |
| cloudnull | ^ sorry fixing lint issue | 03:50 |
| *** gomarivera has quit IRC | 03:50 | |
| ianw | cloudnull: scripts in install.d are running inside a built-image chroot, yep. but they have /proc /sys etc mounted from the build system. | 03:50 |
| cloudnull | so the recent changes should be good right? | 03:51 |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| *** sflanigan has joined #openstack-infra | 03:51 | |
| *** raunak has quit IRC | 03:52 | |
| *** raunak has joined #openstack-infra | 03:53 | |
| *** links has joined #openstack-infra | 03:54 | |
| *** tqtran has joined #openstack-infra | 03:55 | |
| *** yuanying has quit IRC | 03:55 | |
| *** oanson has quit IRC | 03:55 | |
| *** thorst has quit IRC | 03:56 | |
| njohnston | Hi! Sorry to be a pest, but I noticed my change (339246) had been in the gate queue for about 4 hours... and everything was looking fine, jobs were queued... but now it says "CANCELLED". What should I do to kickstart it back in again? And is there something I should wait for before doing so? | 03:56 |
| njohnston | oh, and now it's not cancelled anymore, it's back to queued. So never mind! | 03:57 |
| *** tqtran has quit IRC | 03:59 | |
| *** kbaegis has quit IRC | 04:02 | |
| *** hockeynut has quit IRC | 04:03 | |
| *** coolsvap has joined #openstack-infra | 04:04 | |
| *** amitgandhinz has joined #openstack-infra | 04:04 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Disabled IPv6 privacy extensions https://review.openstack.org/356204 | 04:04 |
| cloudnull | sorry for the spam -- fixed typos | 04:04 |
| *** tphummel has quit IRC | 04:07 | |
| *** amitgandhinz has quit IRC | 04:08 | |
| cloudnull | ianw: interestingly, on a vanilla build w/ the ubuntu cloud image I have "/etc/sysctl.d/99-cloudimg-ipv6.conf" which has http://cdn.pasteraw.com/sevez8vbfdqnuedbycjt9o4oddbcsxl | 04:09 |
| cloudnull | which has https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1068756 | 04:10 |
| openstack | Launchpad bug 1068756 in cloud-init (Ubuntu) "IPv6 Privacy Extensions enabled on Ubuntu Server by default" [Medium,Triaged] | 04:10 |
| *** dimtruck is now known as zz_dimtruck | 04:10 | |
| *** zz_dimtruck is now known as dimtruck | 04:10 | |
| cloudnull | so i think i'm going to link to that bug too, and drop the same file. | 04:10 |
| cloudnull | because thats the issue we're seeing in the gate. | 04:10 |
| *** aeng has quit IRC | 04:12 | |
| *** zhurong has quit IRC | 04:14 | |
| *** zhurong has joined #openstack-infra | 04:15 | |
| *** yuanying has joined #openstack-infra | 04:16 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Disabled IPv6 privacy extensions https://review.openstack.org/356204 | 04:19 |
| *** gomarivera has joined #openstack-infra | 04:19 | |
| *** dimtruck is now known as zz_dimtruck | 04:19 | |
| *** shashank_hegde has joined #openstack-infra | 04:19 | |
| *** apetrich has joined #openstack-infra | 04:22 | |
| *** raunak has quit IRC | 04:23 | |
| *** devkulkarni1 has quit IRC | 04:23 | |
| *** aeng has joined #openstack-infra | 04:25 | |
| *** kcobb has quit IRC | 04:25 | |
| *** kcobb has joined #openstack-infra | 04:26 | |
| *** bswartz has quit IRC | 04:27 | |
| *** kushal has joined #openstack-infra | 04:27 | |
| ianw | cloudnull: cool. might actually something we should consider for dib proper and the minimal builds ... anyway, let's start with that :) | 04:34 |
| cloudnull | indeed. | 04:34 |
| cloudnull | if it makes the gate happy we can go from there. | 04:34 |
| *** raunak has joined #openstack-infra | 04:34 | |
| cloudnull | or, make it happier... | 04:35 |
| cloudnull | nothing makes that gate completely happy ;) | 04:35 |
| *** amotoki has joined #openstack-infra | 04:38 | |
| *** davidlenwell has quit IRC | 04:42 | |
| *** oanson has joined #openstack-infra | 04:47 | |
| *** abregman has joined #openstack-infra | 04:48 | |
| *** davidlenwell has joined #openstack-infra | 04:55 | |
| *** thorst has joined #openstack-infra | 04:55 | |
| *** sflanigan has quit IRC | 04:56 | |
| *** dchen has quit IRC | 04:56 | |
| *** narayrak has joined #openstack-infra | 05:00 | |
| *** thorst has quit IRC | 05:01 | |
| *** pahuang has quit IRC | 05:04 | |
| *** amitgandhinz has joined #openstack-infra | 05:04 | |
| *** sflanigan has joined #openstack-infra | 05:09 | |
| *** sflanigan has joined #openstack-infra | 05:09 | |
| *** amitgandhinz has quit IRC | 05:09 | |
| *** adrian_otto has joined #openstack-infra | 05:10 | |
| *** jaosorior has joined #openstack-infra | 05:13 | |
| *** adrian_otto has quit IRC | 05:14 | |
| *** gomarivera has quit IRC | 05:14 | |
| *** pahuang has joined #openstack-infra | 05:17 | |
| *** itisha has quit IRC | 05:20 | |
| *** jimbaker has quit IRC | 05:22 | |
| *** jimbaker has joined #openstack-infra | 05:26 | |
| *** jimbaker has quit IRC | 05:26 | |
| *** jimbaker has joined #openstack-infra | 05:26 | |
| *** gildub has quit IRC | 05:26 | |
| *** yuanying_ has joined #openstack-infra | 05:29 | |
| *** yuanying has quit IRC | 05:29 | |
| *** ihrachys has joined #openstack-infra | 05:30 | |
| *** abashmak has joined #openstack-infra | 05:31 | |
| abashmak | Hello | 05:32 |
| openstackgerrit | Tang Chen proposed openstack-infra/project-config: Enable ironic functional test https://review.openstack.org/356233 | 05:35 |
| openstackgerrit | Tang Chen proposed openstack-infra/project-config: Enable ironic functional test https://review.openstack.org/356233 | 05:36 |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: test non-default value of attach-build-log https://review.openstack.org/356236 | 05:37 |
| *** sandanar has joined #openstack-infra | 05:37 | |
| *** abashmak has left #openstack-infra | 05:41 | |
| *** rbuzatu has joined #openstack-infra | 05:45 | |
| *** rcernin has joined #openstack-infra | 05:46 | |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: add compress-log option to compress log https://review.openstack.org/354138 | 05:49 |
| *** rbuzatu has quit IRC | 05:50 | |
| *** gildub has joined #openstack-infra | 05:51 | |
| *** _nadya_ has joined #openstack-infra | 05:56 | |
| *** asettle has joined #openstack-infra | 06:00 | |
| *** thorst has joined #openstack-infra | 06:00 | |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: add post-send script option https://review.openstack.org/355135 | 06:03 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Add additional options to unbound https://review.openstack.org/356241 | 06:03 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Add additional options to unbound https://review.openstack.org/356241 | 06:05 |
| *** asettle has quit IRC | 06:05 | |
| *** amitgandhinz has joined #openstack-infra | 06:05 | |
| *** thorst has quit IRC | 06:06 | |
| *** dfflanders has joined #openstack-infra | 06:09 | |
| *** amitgandhinz has quit IRC | 06:10 | |
| *** ihrachys has quit IRC | 06:15 | |
| *** roxanagh_ has joined #openstack-infra | 06:16 | |
| *** Sukhdev has joined #openstack-infra | 06:20 | |
| *** roxanagh_ has quit IRC | 06:22 | |
| *** e0ne has joined #openstack-infra | 06:30 | |
| *** sandanar_ has joined #openstack-infra | 06:32 | |
| *** sandanar has quit IRC | 06:35 | |
| *** adriant has quit IRC | 06:37 | |
| *** raunak has quit IRC | 06:41 | |
| *** caowei has joined #openstack-infra | 06:41 | |
| *** pgadiya has joined #openstack-infra | 06:44 | |
| *** caowei has quit IRC | 06:45 | |
| *** _nadya_ has quit IRC | 06:45 | |
| *** caowei has joined #openstack-infra | 06:45 | |
| *** caowei has quit IRC | 06:45 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 06:46 |
| *** caowei has joined #openstack-infra | 06:46 | |
| *** Sukhdev has quit IRC | 06:46 | |
| *** Jeffrey4l has joined #openstack-infra | 06:49 | |
| *** raunak has joined #openstack-infra | 06:49 | |
| *** javeriak has joined #openstack-infra | 06:50 | |
| *** amotoki has quit IRC | 06:50 | |
| openstackgerrit | Henry Gessau proposed openstack-infra/project-config: Use python-db-jobs for networking-sfc https://review.openstack.org/354358 | 06:52 |
| *** dtardivel has joined #openstack-infra | 06:53 | |
| *** raunak has quit IRC | 06:55 | |
| *** _nadya_ has joined #openstack-infra | 06:57 | |
| openstackgerrit | Joost van der Griendt proposed openstack-infra/jenkins-job-builder: Adding support for Hidden parameter plugin https://review.openstack.org/355209 | 06:58 |
| *** shashank_hegde has quit IRC | 06:58 | |
| *** adrian_otto has joined #openstack-infra | 06:58 | |
| *** sandanar__ has joined #openstack-infra | 06:59 | |
| *** gouthamr has quit IRC | 07:00 | |
| *** markvoelker has quit IRC | 07:00 | |
| *** tesseract- has joined #openstack-infra | 07:00 | |
| *** _nadya_ has quit IRC | 07:00 | |
| yolanda | good morning | 07:01 |
| *** mhickey has joined #openstack-infra | 07:01 | |
| *** sandanar_ has quit IRC | 07:02 | |
| *** raunak has joined #openstack-infra | 07:03 | |
| *** adrian_otto has quit IRC | 07:03 | |
| *** thorst has joined #openstack-infra | 07:03 | |
| *** matbu|afk is now known as matbu | 07:03 | |
| *** amitgandhinz has joined #openstack-infra | 07:06 | |
| *** jimbaker has quit IRC | 07:07 | |
| *** raunak has quit IRC | 07:09 | |
| *** amitgandhinz has quit IRC | 07:11 | |
| *** thorst has quit IRC | 07:11 | |
| *** jimbaker has joined #openstack-infra | 07:11 | |
| *** jimbaker has quit IRC | 07:11 | |
| *** jimbaker has joined #openstack-infra | 07:11 | |
| openstackgerrit | Joost van der Griendt proposed openstack-infra/jenkins-job-builder: adding extra properties to sonar publisher https://review.openstack.org/355206 | 07:13 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Don't use AFS mirrors in no nodepool environment https://review.openstack.org/350003 | 07:15 |
| *** shashank_hegde has joined #openstack-infra | 07:17 | |
| *** shashank_hegde has quit IRC | 07:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Make ironic job non-voting on Neutron https://review.openstack.org/356072 | 07:19 |
| *** Goneri has joined #openstack-infra | 07:20 | |
| *** raunak has joined #openstack-infra | 07:21 | |
| *** jpich has joined #openstack-infra | 07:23 | |
| *** penguinolog__ has joined #openstack-infra | 07:25 | |
| *** andymaier has joined #openstack-infra | 07:26 | |
| *** pt_15 has quit IRC | 07:26 | |
| *** penguinolog__ has quit IRC | 07:26 | |
| *** gildub has quit IRC | 07:29 | |
| *** yamahata has joined #openstack-infra | 07:29 | |
| *** thorst has joined #openstack-infra | 07:29 | |
| *** abregman_ has joined #openstack-infra | 07:31 | |
| *** rbuzatu has joined #openstack-infra | 07:33 | |
| *** abregman has quit IRC | 07:33 | |
| *** thorst has quit IRC | 07:34 | |
| *** esikachev has joined #openstack-infra | 07:34 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Update TLS endpoints https://review.openstack.org/356286 | 07:35 |
| *** florianf has joined #openstack-infra | 07:37 | |
| *** abregman_ is now known as abregman | 07:41 | |
| *** jamielennox is now known as jamielennox|away | 07:45 | |
| *** raunak has quit IRC | 07:45 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 07:45 |
| *** caowei has quit IRC | 07:47 | |
| *** larainema has joined #openstack-infra | 07:48 | |
| *** dtantsur|afk is now known as dtantsur | 07:49 | |
| *** abregman has quit IRC | 07:49 | |
| *** e0ne has quit IRC | 07:51 | |
| *** raunak has joined #openstack-infra | 07:53 | |
| *** javeriak has quit IRC | 07:58 | |
| *** javeriak has joined #openstack-infra | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** matthewbodkin has joined #openstack-infra | 08:00 | |
| *** markvoelker has joined #openstack-infra | 08:00 | |
| *** jimbaker has quit IRC | 08:01 | |
| *** jed56 has joined #openstack-infra | 08:01 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** abregman has joined #openstack-infra | 08:01 | |
| *** amotoki has joined #openstack-infra | 08:02 | |
| *** nmagnezi has joined #openstack-infra | 08:02 | |
| *** amotoki has quit IRC | 08:02 | |
| *** amotoki has joined #openstack-infra | 08:03 | |
| *** jimbaker has joined #openstack-infra | 08:05 | |
| *** jimbaker has quit IRC | 08:05 | |
| *** jimbaker has joined #openstack-infra | 08:05 | |
| *** markvoelker has quit IRC | 08:05 | |
| *** ihrachys has joined #openstack-infra | 08:05 | |
| *** kushal has quit IRC | 08:06 | |
| *** amitgandhinz has joined #openstack-infra | 08:07 | |
| *** martinkopec has joined #openstack-infra | 08:07 | |
| *** ccamacho has joined #openstack-infra | 08:08 | |
| *** amotoki_ has joined #openstack-infra | 08:08 | |
| *** amotoki has quit IRC | 08:08 | |
| *** coolsvap is now known as coolsvap_ | 08:09 | |
| *** raunak has quit IRC | 08:09 | |
| *** chem has joined #openstack-infra | 08:09 | |
| *** armax has joined #openstack-infra | 08:09 | |
| *** abregman is now known as abregman|mtg | 08:11 | |
| *** amitgandhinz has quit IRC | 08:11 | |
| *** raunak has joined #openstack-infra | 08:14 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard: Fix docs so it is easy to understand https://review.openstack.org/355886 | 08:14 |
| *** martinkopec has quit IRC | 08:16 | |
| *** yaume has joined #openstack-infra | 08:17 | |
| *** dingyichen has quit IRC | 08:17 | |
| *** lucas-dinner is now known as lucasagomes | 08:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Rename bifrost integration test job https://review.openstack.org/355652 | 08:18 |
| *** martinkopec has joined #openstack-infra | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Make the grenade job voting on ironic-inspector https://review.openstack.org/355894 | 08:19 |
| *** e0ne has joined #openstack-infra | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Add release notes jobs for ironic-ui https://review.openstack.org/356029 | 08:21 |
| openstackgerrit | Merged openstack-infra/project-config: Add ansible-role-jobs for browbeat https://review.openstack.org/356093 | 08:21 |
| openstackgerrit | Merged openstack-infra/project-config: Use python-db-jobs for tap-as-a-service https://review.openstack.org/355670 | 08:21 |
| dtantsur | hi folks! please review https://review.openstack.org/#/c/355881/ it's one more thing blocking ironic stable gate. thanks! | 08:22 |
| *** infra-red has joined #openstack-infra | 08:22 | |
| *** asettle has joined #openstack-infra | 08:22 | |
| *** rlandy has quit IRC | 08:24 | |
| *** priteau has joined #openstack-infra | 08:24 | |
| *** acoles_ is now known as acoles | 08:24 | |
| *** matrohon has joined #openstack-infra | 08:26 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Enable create_ipa_image flag https://review.openstack.org/356305 | 08:26 |
| *** ihrachys has quit IRC | 08:29 | |
| *** thorst has joined #openstack-infra | 08:31 | |
| tlbr | hi, could please someone review https://review.openstack.org/347047 ? | 08:32 |
| *** dkehn has quit IRC | 08:32 | |
| *** dkehn_ has quit IRC | 08:32 | |
| *** ihrachys has joined #openstack-infra | 08:34 | |
| *** Jaison has joined #openstack-infra | 08:35 | |
| *** links has quit IRC | 08:37 | |
| *** thorst has quit IRC | 08:38 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 08:38 |
| *** dkehn has joined #openstack-infra | 08:38 | |
| dtantsur | folks, any news on check-osc-plugins instability? I'm afraid we'll have to drop it from voting temporary unless it's fixed soon... | 08:41 |
| *** _nadya_ has joined #openstack-infra | 08:42 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Disable DNS resolver on Bifrost dnsmasq server https://review.openstack.org/355973 | 08:42 |
| *** adrian_otto has joined #openstack-infra | 08:47 | |
| openstackgerrit | Tom Fifield proposed openstack-infra/puppet-mediawiki: Add 'autopatrol' group https://review.openstack.org/356313 | 08:48 |
| *** jamielennox|away is now known as jamielennox | 08:50 | |
| *** adrian_otto has quit IRC | 08:51 | |
| *** dkehn_ has joined #openstack-infra | 08:52 | |
| *** javeriak has quit IRC | 08:52 | |
| *** _degorenko|afk is now known as degorenko | 08:52 | |
| *** elo has quit IRC | 08:52 | |
| *** xarses_ has joined #openstack-infra | 08:55 | |
| *** derekh has joined #openstack-infra | 08:57 | |
| *** ihrachys has quit IRC | 08:57 | |
| *** xarses has quit IRC | 08:58 | |
| *** asettle has quit IRC | 08:59 | |
| *** raunak has quit IRC | 08:59 | |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/zuul: Set keepalives for gerrit connections https://review.openstack.org/238988 | 09:01 |
| *** markvoelker has joined #openstack-infra | 09:01 | |
| *** asettle has joined #openstack-infra | 09:01 | |
| *** asettle has quit IRC | 09:02 | |
| *** elo has joined #openstack-infra | 09:02 | |
| *** asettle has joined #openstack-infra | 09:03 | |
| *** ihrachys has joined #openstack-infra | 09:03 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard: Fix docs so it is easy to understand https://review.openstack.org/355886 | 09:04 |
| *** armax has quit IRC | 09:06 | |
| *** markvoelker has quit IRC | 09:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Prevent Ironic multitenancy job running on old versions https://review.openstack.org/355881 | 09:07 |
| *** armax has joined #openstack-infra | 09:07 | |
| *** amitgandhinz has joined #openstack-infra | 09:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Add searchlight-ui translation https://review.openstack.org/353079 | 09:08 |
| *** raunak has joined #openstack-infra | 09:10 | |
| *** ihrachys has quit IRC | 09:11 | |
| *** ihrachys has joined #openstack-infra | 09:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish manila install guide from tree https://review.openstack.org/352173 | 09:11 |
| *** elo has quit IRC | 09:12 | |
| *** amitgandhinz has quit IRC | 09:12 | |
| *** elo has joined #openstack-infra | 09:13 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:13 | |
| *** abregman|mtg is now known as abregman | 09:15 | |
| *** raunak has quit IRC | 09:15 | |
| *** tqtran has joined #openstack-infra | 09:19 | |
| *** asettle has quit IRC | 09:20 | |
| *** gus has quit IRC | 09:22 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 09:22 |
| *** kushal has joined #openstack-infra | 09:23 | |
| *** tqtran has quit IRC | 09:23 | |
| *** andymaier has quit IRC | 09:23 | |
| *** yamahata has quit IRC | 09:24 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Fix typo in test_attr_list handling https://review.openstack.org/355385 | 09:25 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting https://review.openstack.org/355393 | 09:25 |
| *** matthewbodkin has quit IRC | 09:26 | |
| *** sambetts|afk is now known as sambetts | 09:26 | |
| *** ihrachys has quit IRC | 09:28 | |
| openstackgerrit | Bob Ball proposed openstack-infra/irc-meetings: Update XenAPI meeting room to avoid conflict https://review.openstack.org/356332 | 09:29 |
| *** raunak has joined #openstack-infra | 09:30 | |
| *** ihrachys has joined #openstack-infra | 09:30 | |
| *** matthewbodkin has joined #openstack-infra | 09:32 | |
| *** Hal has joined #openstack-infra | 09:33 | |
| *** javeriak has joined #openstack-infra | 09:35 | |
| *** javeriak_ has joined #openstack-infra | 09:35 | |
| *** asettle has joined #openstack-infra | 09:36 | |
| *** javeriak has quit IRC | 09:36 | |
| *** javeriak has joined #openstack-infra | 09:36 | |
| *** thorst has joined #openstack-infra | 09:37 | |
| openstackgerrit | Merged openstack-infra/system-config: Move other-requirements.txt to bindep.txt https://review.openstack.org/354868 | 09:37 |
| *** asettle has quit IRC | 09:38 | |
| *** electrofelix has joined #openstack-infra | 09:39 | |
| *** javeriak_ has quit IRC | 09:40 | |
| *** tosky has joined #openstack-infra | 09:41 | |
| *** jamielennox is now known as jamielennox|away | 09:43 | |
| *** thorst has quit IRC | 09:44 | |
| *** spzala has joined #openstack-infra | 09:45 | |
| *** elo has quit IRC | 09:46 | |
| *** javeriak has quit IRC | 09:47 | |
| *** mrunge has quit IRC | 09:49 | |
| *** spzala has quit IRC | 09:49 | |
| *** elo has joined #openstack-infra | 09:50 | |
| *** andymaier has joined #openstack-infra | 09:53 | |
| *** gomarivera has joined #openstack-infra | 09:54 | |
| *** javeriak has joined #openstack-infra | 09:55 | |
| *** jimbaker has quit IRC | 09:55 | |
| *** mrunge has joined #openstack-infra | 09:55 | |
| *** alexz__ has joined #openstack-infra | 09:55 | |
| *** elo has quit IRC | 09:55 | |
| *** openstack has joined #openstack-infra | 10:16 | |
| *** dkehn has joined #openstack-infra | 10:17 | |
| Zara | and the bot has come back here, too \o/ | 10:17 |
| oneswig | Fixed - thanks, you don't need to admit to that :-) | 10:17 |
| rcarrillocruz | we have autohealing infra :P | 10:17 |
| openstackgerrit | Merged openstack-infra/system-config: Correct public IP for baremetal00 https://review.openstack.org/355841 | 10:17 |
| *** sdake_ has quit IRC | 10:18 | |
| *** e0ne has quit IRC | 10:18 | |
| *** vgridnev has joined #openstack-infra | 10:18 | |
| *** oneswig has quit IRC | 10:18 | |
| *** sdake_ has joined #openstack-infra | 10:19 | |
| *** sdake has quit IRC | 10:19 | |
| vgridnev | tosky, about your message, you probably will need help of infra-root | 10:19 |
| vgridnev | SergeyLukjanov is not just infra-core | 10:20 |
| tosky | vgridnev: I know, I mentioned him just to explain that it was done in the past but he is not available now | 10:21 |
| vgridnev | ok | 10:21 |
| vgridnev | thanks | 10:22 |
| *** dkehn_ has joined #openstack-infra | 10:22 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Enable create_ipa_image flag https://review.openstack.org/356305 | 10:22 |
| vgridnev | I've actually forwarded that message to him but seems that I did not received any answer | 10:22 |
| vgridnev | anyway I think that we can't start the process of final migration; and then we will have all tests in one place \o/ | 10:25 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 10:26 |
| vgridnev | we can't -> we can | 10:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Create timeline events for boards and worklists https://review.openstack.org/350146 | 10:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 10:26 |
| *** sdague has joined #openstack-infra | 10:27 | |
| *** ihrachys has quit IRC | 10:30 | |
| *** psachin has joined #openstack-infra | 10:31 | |
| *** gomarivera has joined #openstack-infra | 10:32 | |
| *** adrian_otto has joined #openstack-infra | 10:35 | |
| *** gomarivera has quit IRC | 10:37 | |
| *** adrian_otto has quit IRC | 10:40 | |
| *** raunak has quit IRC | 10:42 | |
| *** thorst has joined #openstack-infra | 10:42 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-infracloud: Remove addressing_mode line from global vars to set static IP on config drives https://review.openstack.org/356373 | 10:43 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Set Mistral workers to 1 https://review.openstack.org/356375 | 10:45 |
| yolanda | infra-root, when you come online... there is a recurring problem on osic, on ubuntu-trusty | 10:47 |
| yolanda | i'm hitting that error constantly https://review.openstack.org/#/c/356373/1 ... puppet apply test crashes suddenly when cloning puppet-hound | 10:48 |
| *** amitgandhinz has joined #openstack-infra | 10:48 | |
| yolanda | it is passing on other providers, and i double checked puppet-hound repo and everything looks fine | 10:48 |
| *** thorst has quit IRC | 10:48 | |
| sdague | yolanda: fungi and others were looking at it yesterday. The last I knew it was osic + trusty + ipv6 routing not working | 10:49 |
| *** Hal has quit IRC | 10:49 | |
| yolanda | shall we disable osic? | 10:49 |
| sdague | maybe? can we disable it for only trusty nodes? | 10:49 |
| *** jimbaker has quit IRC | 10:50 | |
| rcarrillocruz | i guess we could remove the trusty flavor from the osic provider | 10:50 |
| yolanda | yes, i think so | 10:50 |
| * rcarrillocruz goes check | 10:50 | |
| *** ociuhandu has quit IRC | 10:50 | |
| yolanda | rcarrillocruz, so if you want to create a change, i'll review | 10:50 |
| *** yamamoto has joined #openstack-infra | 10:50 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Disable trusty nodes on OSIC https://review.openstack.org/356382 | 10:51 |
| rcarrillocruz | sdague, yolanda ^ | 10:51 |
| sdague | +2 | 10:52 |
| *** amitgandhinz has quit IRC | 10:52 | |
| yolanda | approved | 10:52 |
| *** gildub has joined #openstack-infra | 10:53 | |
| *** jimbaker has joined #openstack-infra | 10:54 | |
| *** jimbaker has quit IRC | 10:54 | |
| *** jimbaker has joined #openstack-infra | 10:54 | |
| *** yanyanhu has quit IRC | 10:54 | |
| *** asettle has joined #openstack-infra | 10:55 | |
| *** Guest26397 has joined #openstack-infra | 10:56 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:57 | |
| amrith | sdague, have a second to spare. need some help with the api-ref code that I think you wrote. would you take a look at https://review.openstack.org/#/c/316381/, the failing job gate-trove-tox-doc-publish-checkbuild (logs at http://logs.openstack.org/81/316381/14/check/gate-trove-tox-doc-publish-checkbuild/b9a3613/). If you look at the tox.ini file the change has a comment with your name on it. https://review.openstack.or | 10:58 |
| amrith | g/#/c/316381/14/tox.ini. Would you help me understand that please. The comment and the code don't seem to be in sync. | 10:58 |
| amrith | sdague ^^ | 10:59 |
| *** andymaier has quit IRC | 11:00 | |
| sdague | I'm pretty sure everything supports constraints now | 11:00 |
| amrith | I thought so sdague and I removed the install_command; it failed identically | 11:01 |
| sdague | that patch originated may 14th, so I'm not surprised that is out of date | 11:01 |
| amrith | see https://review.openstack.org/#/c/316381/11 | 11:02 |
| amrith | :) | 11:02 |
| amrith | another thing, the comment in tox.ini about not using -W, what is that about? | 11:02 |
| *** yamamoto has quit IRC | 11:02 | |
| amrith | I tried to see what other projects are doing | 11:02 |
| sdague | this is what happens when people take the nova tox.ini and just copy paste it elsewhere | 11:03 |
| sdague | you can't enforce -W until your source is clean | 11:03 |
| sdague | ours wasn't for a while | 11:03 |
| *** Hal has joined #openstack-infra | 11:03 | |
| amrith | ok, so this is copy-pasta that I'm dealing with | 11:03 |
| sdague | yeh | 11:03 |
| amrith | will make some of my own pasta. | 11:03 |
| sdague | heh | 11:03 |
| *** yamamoto has joined #openstack-infra | 11:03 | |
| amrith | thx sdague the upper-constraints stuff doesn't ring a bell. will investigate. thx muc | 11:04 |
| amrith | much | 11:04 |
| sdague | I'm not an expert on the publish build there, maybe yolanda or others here could help | 11:04 |
| amrith | sdague, this works locally (the publish part) | 11:05 |
| amrith | the thing I need to figure out now is what ails the upper-constraints.txt thing | 11:05 |
| amrith | because at this point, sphinx isn't even running in the CI | 11:05 |
| amrith | and this particular upper-constraints.txt error is one I've not seen before. | 11:05 |
| *** kushal has quit IRC | 11:06 | |
| sdague | it may be part of trove specific job configs, I don't really know though | 11:06 |
| amrith | that it may well be; I'm not positive either though none of the other jobs appear to see this. Will dig ... | 11:07 |
| *** yamamoto has quit IRC | 11:07 | |
| *** rbuzatu has quit IRC | 11:07 | |
| *** yamamoto has joined #openstack-infra | 11:07 | |
| *** gongysh has quit IRC | 11:08 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update TLS endpoints https://review.openstack.org/356286 | 11:08 |
| *** yamamoto has quit IRC | 11:09 | |
| *** yamamoto has joined #openstack-infra | 11:09 | |
| *** Guest26397 is now known as reasonal78 | 11:10 | |
| *** dtantsur is now known as dtantsur|bbl | 11:11 | |
| *** gomarivera has joined #openstack-infra | 11:14 | |
| *** matbu is now known as matbu|lch | 11:16 | |
| *** thorst has joined #openstack-infra | 11:17 | |
| *** andymaier has joined #openstack-infra | 11:19 | |
| *** gomarivera has quit IRC | 11:19 | |
| *** thorst_ has joined #openstack-infra | 11:21 | |
| *** ihrachys has joined #openstack-infra | 11:22 | |
| *** lucasagomes is now known as lucas-hungry | 11:23 | |
| *** thorst has quit IRC | 11:23 | |
| *** vgridnev has quit IRC | 11:24 | |
| *** sdake_ is now known as sdake | 11:24 | |
| *** yaume has quit IRC | 11:26 | |
| *** vgridnev has joined #openstack-infra | 11:27 | |
| amrith | sdague, the error may be because of https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L2671-L2679 | 11:27 |
| amrith | this appears to be some kind of a kludge that should no longer be there. | 11:27 |
| amrith | fungi, yolanda would you help me understand this please. | 11:27 |
| amrith | am trying to figure out why I'm getting this failure in https://review.openstack.org/#/c/316381 | 11:27 |
| *** ociuhandu has joined #openstack-infra | 11:28 | |
| amrith | the job in question in gate-trove-tox-doc-publish-checkbuild | 11:28 |
| yolanda | amrith, not an expert on that, but going to take a look | 11:28 |
| amrith | The failure is http://logs.openstack.org/81/316381/14/check/gate-trove-tox-doc-publish-checkbuild/b9a3613/ | 11:28 |
| amrith | thx yolanda | 11:28 |
| amrith | the last link above | 11:28 |
| amrith | the output from the job | 11:28 |
| amrith | is very weird | 11:28 |
| amrith | and it appears that trove has this one little thing that no one else has in layout.yaml | 11:28 |
| *** jkilpatr has joined #openstack-infra | 11:28 | |
| yolanda | Could not open requirements file: [Errno 2] No such file or directory: '/home/jenkins/workspace/gate-trove-tox-doc-publish-checkbuild/upper-constraints.txt' | 11:28 |
| amrith | https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L2671-L2679 | 11:29 |
| amrith | yes, that error | 11:29 |
| *** jianghuaw has joined #openstack-infra | 11:30 | |
| *** ramishra has quit IRC | 11:30 | |
| sdague | amrith: trove appears to be the only project with that job | 11:31 |
| *** lbeliveau has quit IRC | 11:31 | |
| amrith | not really, there's the job openstack-doc-gate | 11:31 |
| amrith | which includes gate-{name}-tox-doc-publish-checkbuild | 11:31 |
| amrith | but do I understand correctly that this job | 11:32 |
| amrith | gate-trove-tox-doc-publish-checbuild | 11:32 |
| amrith | is merely being restricted in those lines | 11:32 |
| amrith | https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L2671-L2679 | 11:32 |
| *** ramishra has joined #openstack-infra | 11:32 | |
| *** ihrachys has quit IRC | 11:32 | |
| amrith | i.e all it says is that there's this job, only run it if stuff in this dir changes? | 11:32 |
| sdague | https://github.com/openstack-infra/project-config/blob/5d5c4a25803f8566fc222df00e18b6ebd02fe253/zuul/layout.yaml#L9544 | 11:32 |
| amrith | or is it defining a net new job? | 11:32 |
| sdague | amrith: openstack-doc-gate is a radically different thing | 11:32 |
| sdague | the builder for api-ref is api-ref-jobs | 11:33 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 11:33 |
| amrith | and also trove has no api-ref job https://github.com/openstack-infra/project-config/blob/5d5c4a25803f8566fc222df00e18b6ebd02fe253/zuul/layout.yaml#L14274-L14306 | 11:34 |
| sdague | http://logs.openstack.org/81/316381/14/check/gate-trove-tox-doc-publish-checkbuild/b9a3613/console.html#_2016-08-17_02_23_29_515859 | 11:34 |
| amrith | ok, so I should add the api-ref job at the very minimum | 11:34 |
| sdague | you are failing when running: tox -e checkbuild | 11:34 |
| amrith | awesome | 11:35 |
| amrith | that was the thing I was missing | 11:35 |
| amrith | let me see what that is | 11:35 |
| sdague | that's completely unrelated to api-ref jobs | 11:35 |
| *** jaosorior has quit IRC | 11:35 | |
| amrith | and I bet that job doesn't know what upper constraints.txt | 11:36 |
| amrith | is | 11:36 |
| amrith | and recently when we cleaned up our tox.ini | 11:36 |
| amrith | the special install_command for that must've been removed | 11:36 |
| amrith | I bet | 11:36 |
| amrith | let me check | 11:36 |
| *** jaosorior has joined #openstack-infra | 11:36 | |
| amrith | Yup! thanks sdague I can fix this | 11:37 |
| amrith | yolanda, I think sdague just explained what's wrong | 11:37 |
| *** lbeliveau has joined #openstack-infra | 11:37 | |
| amrith | let me try to re-run the gate with this change | 11:37 |
| amrith | and see how it works out | 11:37 |
| yolanda | ah the upper failure right? i was looking but logs are not published | 11:37 |
| yolanda | can you verify that locally? | 11:37 |
| amrith | yes I can | 11:37 |
| amrith | the issue is unrelated to this specific change | 11:37 |
| amrith | there's another job (checkbuild) | 11:37 |
| amrith | that would only run if apidocs are changed | 11:38 |
| amrith | and that never does | 11:38 |
| amrith | and so another change ... | 11:38 |
| amrith | git diff 54c4a31b3d2461f87d868d5ab68dd5d087f2f850..5510241c31a45ea735ad0d1a50c29d91c37a8297 | 11:38 |
| amrith | introduced this problem | 11:38 |
| yolanda | ok glad that sdague helped | 11:39 |
| sdague | amrith: right, in all those restricting changes you also need to make them run on tox.ini changes | 11:39 |
| sdague | otherwise you can get into this situation | 11:39 |
| amrith | sdague, YES! | 11:40 |
| amrith | otherwise you can end up planting the seeds for a later explosion (as we just did) | 11:40 |
| *** yamamoto has quit IRC | 11:40 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar https://review.openstack.org/355554 | 11:40 |
| amrith | thx sdague yolanda I just pushed another change set, let's see if it passes muster. | 11:41 |
| amrith | I'll also push a change for project-config | 11:41 |
| *** kbaegis has joined #openstack-infra | 11:42 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-openstackci: Pass Java arguments to Jenkins master https://review.openstack.org/263021 | 11:43 |
| *** ganesan has joined #openstack-infra | 11:43 | |
| sdague | amrith: cool | 11:43 |
| sdague | yeh, definitely add the api-ref jobs to your project, then you'll get building of the api-ref in CI, you aren't getting that at all yet | 11:44 |
| *** javeriak has quit IRC | 11:44 | |
| amrith | I guess that change can only merge after the changes to trove merge (to define the job, for example) | 11:44 |
| jianghuaw | Hi all. I have a special case in CI testing environment: for each job, it requires a pair of VMs which are created from different images. But it seems multi-node in nodepool only supports multiple nodes with the same image. Any advice to achieve my purpose? Thanks a lot for any input. | 11:45 |
| *** sshnaidm is now known as sshnaidm|afk | 11:46 | |
| *** yamamoto has joined #openstack-infra | 11:46 | |
| jianghuaw | My CI environment is setup with zuul + jenkins + nodepool. | 11:46 |
| *** psachin has quit IRC | 11:47 | |
| *** weshay has joined #openstack-infra | 11:48 | |
| *** psachin has joined #openstack-infra | 11:48 | |
| *** ansmith has joined #openstack-infra | 11:49 | |
| *** alexz__ has quit IRC | 11:49 | |
| *** claudiub has joined #openstack-infra | 11:49 | |
| *** claudiub|2 has joined #openstack-infra | 11:49 | |
| *** alexz__ has joined #openstack-infra | 11:49 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Update trove jobs to include api-ref job https://review.openstack.org/356407 | 11:51 |
| *** vgridnev has quit IRC | 11:52 | |
| *** gomarivera has joined #openstack-infra | 11:52 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-openstackci: Pass Java arguments to Jenkins master https://review.openstack.org/263021 | 11:54 |
| *** vgridnev has joined #openstack-infra | 11:54 | |
| *** gomarivera has quit IRC | 11:56 | |
| _nadya_ | hello folks! Could I ask someone to take a look into https://review.openstack.org/#/c/355406/ ? It has one +2 already | 11:56 |
| openstackgerrit | Merged openstack-infra/project-config: Disable trusty nodes on OSIC https://review.openstack.org/356382 | 11:56 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard: Fix docs so it is easy to understand https://review.openstack.org/355886 | 11:57 |
| *** ldnunes has joined #openstack-infra | 11:57 | |
| rcarrillocruz | sdague, yolanda : it merged ^ | 11:58 |
| rcarrillocruz | fungi: head-up in case i'm not around when you start, we disabled OSIC trusty nodes to avoid check/gate failures | 11:58 |
| yolanda | meeting | 11:59 |
| rcarrillocruz | s/head-up/heads-up | 11:59 |
| sdague | rcarrillocruz: thanks | 12:00 |
| *** julim has joined #openstack-infra | 12:01 | |
| *** sdake_ has joined #openstack-infra | 12:01 | |
| tlbr | could please someone review https://review.openstack.org/347047 ? | 12:02 |
| *** kgiusti has joined #openstack-infra | 12:02 | |
| *** andymaier has quit IRC | 12:02 | |
| *** sheeprine has quit IRC | 12:03 | |
| *** coolsvap_ is now known as coolsvap | 12:03 | |
| *** baoli has joined #openstack-infra | 12:03 | |
| *** gouthamr has joined #openstack-infra | 12:04 | |
| *** tpsilva has joined #openstack-infra | 12:04 | |
| *** sdake has quit IRC | 12:05 | |
| *** sdake_ is now known as sdake | 12:05 | |
| jianghuaw | Could anyone help share ideas on my demand to run jobs on a pair of nodes which are created basing on two different images? Is it supported by nodepool or jenkins? | 12:06 |
| *** sheeprine has joined #openstack-infra | 12:06 | |
| *** alexz__ has quit IRC | 12:06 | |
| *** rhallisey has joined #openstack-infra | 12:06 | |
| *** alexz__ has joined #openstack-infra | 12:07 | |
| *** mtanino has joined #openstack-infra | 12:07 | |
| *** rbuzatu has joined #openstack-infra | 12:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Add fuel-plugin-openstack-telemetry https://review.openstack.org/355406 | 12:08 |
| *** baoli has quit IRC | 12:08 | |
| *** rhallisey_ has joined #openstack-infra | 12:08 | |
| *** baoli has joined #openstack-infra | 12:08 | |
| _nadya_ | thanks! | 12:09 |
| *** markvoelker has joined #openstack-infra | 12:09 | |
| *** baoli_ has joined #openstack-infra | 12:09 | |
| *** psachin has quit IRC | 12:10 | |
| vgridnev | hello folks, could you please review https://review.openstack.org/#/c/354700/ ? | 12:11 |
| vgridnev | thanks in advance | 12:11 |
| *** roky has joined #openstack-infra | 12:11 | |
| roky | saluta tutti sul chan | #openstack-infra | | 12:11 |
| *** florianf has quit IRC | 12:11 | |
| *** jamielennox|away is now known as jamielennox | 12:11 | |
| jianghuaw | If jenkins/nodepool don't support multiple-nodes with different images, does zuul-v.3 support the case? I hope jenkins/nodepool could support it. but it's also good if that's supported in zuul-v3:-) | 12:12 |
| jroll | hello roky :) | 12:12 |
| *** dfflanders has quit IRC | 12:12 | |
| *** sarob has joined #openstack-infra | 12:12 | |
| *** rhallisey has quit IRC | 12:12 | |
| *** amotoki_ has quit IRC | 12:13 | |
| *** florianf has joined #openstack-infra | 12:13 | |
| *** rhallisey_ is now known as rhallisey | 12:13 | |
| rcarrillocruz | jianghuaw: i believe it should https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 12:14 |
| *** baoli has quit IRC | 12:14 | |
| *** rbuzatu has quit IRC | 12:14 | |
| *** rodrigods has quit IRC | 12:14 | |
| rcarrillocruz | best ask jeblair when he's around | 12:14 |
| *** rodrigods has joined #openstack-infra | 12:14 | |
| *** andymaier has joined #openstack-infra | 12:14 | |
| *** yamamoto has quit IRC | 12:15 | |
| jianghuaw | rcarrillocruz: thanks. | 12:15 |
| jianghuaw | yes, it seems zuulv3 supports that case. | 12:16 |
| *** elo has joined #openstack-infra | 12:16 | |
| *** dprince has joined #openstack-infra | 12:16 | |
| *** sandanar__ has quit IRC | 12:17 | |
| *** sarob has quit IRC | 12:17 | |
| *** roky has quit IRC | 12:17 | |
| ganesan | I see SSH expection on zuul.log | 12:17 |
| ganesan | SSHException: not a valid EC private key file | 12:17 |
| *** roky has joined #openstack-infra | 12:17 | |
| *** roky has quit IRC | 12:18 | |
| jianghuaw | But i'm still wondering if it's possible to beachieved with Jenkins + nodepool + zuul 2.x. | 12:18 |
| ganesan | can anyone help me what is wrong with my CI system | 12:18 |
| *** chem has quit IRC | 12:18 | |
| *** chem has joined #openstack-infra | 12:19 | |
| *** pilgrimstack has joined #openstack-infra | 12:19 | |
| jianghuaw | We may upgrade to zuul3 in the future, but currently our CI is still running with nodepool + zuul2.1. | 12:19 |
| openstackgerrit | Alexey Ovchinnikov proposed openstack-infra/project-config: [Manila] Move container job to check https://review.openstack.org/356421 | 12:19 |
| *** amotoki has joined #openstack-infra | 12:19 | |
| *** matbu|lch is now known as matbu | 12:20 | |
| *** nwkarste_ has joined #openstack-infra | 12:22 | |
| rcarrillocruz | ganesan: you must be sure zuul user is able to access the private key file | 12:22 |
| rcarrillocruz | i.e. perms | 12:22 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 12:24 |
| *** nwkarste_ has quit IRC | 12:26 | |
| *** lucas-hungry is now known as lucasagomes | 12:26 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 12:27 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Create timeline events for boards and worklists https://review.openstack.org/350146 | 12:27 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 12:28 |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| *** e0ne has joined #openstack-infra | 12:28 | |
| ganesan | rcarrillocuruz: I generated ssh keys using this command "ssh-keygen -t rsa -b 1024 -N '' -f gerrit_key" "ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key"and pasted in /ect/puppet/environments/common.yaml file | 12:28 |
| *** weshay is now known as weshay_mtg | 12:28 | |
| ganesan | and applied puppet | 12:28 |
| *** nwkarsten has joined #openstack-infra | 12:28 | |
| *** notnownikki has joined #openstack-infra | 12:28 | |
| *** raildo has joined #openstack-infra | 12:29 | |
| *** yamamoto has quit IRC | 12:32 | |
| rcarrillocruz | right | 12:32 |
| rcarrillocruz | try accessing manually with zuul user with that key | 12:33 |
| rcarrillocruz | that should give you pointers on what's up | 12:33 |
| rcarrillocruz | k, going for lunch | 12:36 |
| rcarrillocruz | fungi: despite the osic trusty disabling, i haven't deleted manually nodes that were currently available | 12:36 |
| rcarrillocruz | just in case you wanted to hold something to test further what's up | 12:36 |
| rcarrillocruz | later | 12:36 |
| *** claudiub|3 has joined #openstack-infra | 12:37 | |
| hughsaunders | Hey do infra still use jenkins-jclouds at all? | 12:37 |
| *** claudiub|4 has joined #openstack-infra | 12:37 | |
| rcarrillocruz | hughsaunders: i believe that was long abandoned, we used python-jenkins for jjb | 12:38 |
| mordred | hughsaunders: nope. we actually never wound up using it in prod | 12:38 |
| rcarrillocruz | and now not even | 12:38 |
| rcarrillocruz | due to zuul | 12:38 |
| *** claudiub|2 has quit IRC | 12:38 | |
| *** claudiub has quit IRC | 12:38 | |
| rcarrillocruz | ah, jclouds for creating nodes? | 12:38 |
| rcarrillocruz | hmm, interesting, we use nodepool from as long as i can remember | 12:39 |
| hughsaunders | mordred: Ah ok, it seems to fail on newer versions of bouncy castle, but if it was never used, thats probably not an interesting problem to you | 12:39 |
| rcarrillocruz | (now really out to lunch) | 12:39 |
| hughsaunders | rcarrillocruz: yeah, and enjoy lunch | 12:39 |
| mordred | hughsaunders: yah- we basically caused the first version to be written, but we could never get it stable enough so we wrote nodepool | 12:39 |
| openstackgerrit | Merged openstack-infra/project-config: Disabled IPv6 privacy extensions https://review.openstack.org/356204 | 12:40 |
| *** pilgrimstack has quit IRC | 12:40 | |
| *** devkulkarni has joined #openstack-infra | 12:41 | |
| *** gomarivera has joined #openstack-infra | 12:42 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Remove addressing_mode line from global vars to set static IP on config drives https://review.openstack.org/356373 | 12:44 |
| *** ganesan has quit IRC | 12:45 | |
| *** pilgrimstack has joined #openstack-infra | 12:46 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add recent events templates for worklist notifications https://review.openstack.org/356441 | 12:46 |
| *** tosky has quit IRC | 12:47 | |
| *** gomarivera has quit IRC | 12:47 | |
| *** rlandy has joined #openstack-infra | 12:47 | |
| jianghuaw | All, is zuul-v3 ready for CI deployment? thanks. | 12:48 |
| *** bswartz has joined #openstack-infra | 12:48 | |
| *** _ari_ has joined #openstack-infra | 12:48 | |
| *** mtanino has quit IRC | 12:49 | |
| *** weshay_mtg is now known as weshay | 12:49 | |
| *** tosky has joined #openstack-infra | 12:49 | |
| mordred | jianghuaw: nope | 12:50 |
| mordred | jianghuaw: still working on it | 12:50 |
| *** thorst_ has quit IRC | 12:50 | |
| *** javeriak has joined #openstack-infra | 12:51 | |
| hughsaunders | mordred: haha, I'm coming to that realisation myself, x years behind ;-) | 12:52 |
| *** sshnaidm|afk is now known as sshnaidm | 12:52 | |
| mordred | :) | 12:54 |
| mordred | hughsaunders: sorry ... we did not have fun when we came to that realization, so I'm sorry you're also having it :) | 12:54 |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| *** reasonal78 has quit IRC | 12:57 | |
| *** burgerk has joined #openstack-infra | 12:57 | |
| *** vikrant has quit IRC | 12:59 | |
| vgridnev | one more try: hello folks, could you please review https://review.openstack.org/#/c/354700/ ? | 12:59 |
| vgridnev | thanks in advance | 13:00 |
| vgridnev | there is +2 already | 13:00 |
| *** Julien-zte has joined #openstack-infra | 13:00 | |
| *** spzala_ has joined #openstack-infra | 13:01 | |
| cloudnull | mornings | 13:01 |
| *** psilvad has joined #openstack-infra | 13:01 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: StoryBoard 'about' page tweak https://review.openstack.org/354570 | 13:01 |
| *** claudiub has joined #openstack-infra | 13:01 | |
| *** claudiub|2 has joined #openstack-infra | 13:01 | |
| cloudnull | mornings | 13:02 |
| mordred | morning cloudnull ! | 13:02 |
| *** claudiub|3 has quit IRC | 13:02 | |
| *** claudiub|4 has quit IRC | 13:03 | |
| *** claudiub|3 has joined #openstack-infra | 13:04 | |
| *** claudiub|4 has joined #openstack-infra | 13:04 | |
| cloudnull | yolanda sdague rcarrillocruz: I'm hoping many of the OSIC + Trusty failues can be resolved when https://review.openstack.org/#/c/356204/ is part of the image. | 13:05 |
| *** zhurong has joined #openstack-infra | 13:05 | |
| *** gordc has joined #openstack-infra | 13:05 | |
| cloudnull | :'( sad to see it disabled for now but i understand | 13:05 |
| cloudnull | we'll get there. | 13:05 |
| *** pradk has quit IRC | 13:06 | |
| *** xyang1 has joined #openstack-infra | 13:06 | |
| *** claudiub has quit IRC | 13:06 | |
| Jeffrey4l | anyone check this? ubuntu-xenial-nv jobs are running on stable/mitaka branch. https://review.openstack.org/356311 | 13:06 |
| *** claudiub|2 has quit IRC | 13:06 | |
| sdague | cloudnull: well it was completely ruining the gate | 13:06 |
| sdague | it should have been shut off as soon as we narrowed it down | 13:06 |
| *** jheroux has joined #openstack-infra | 13:07 | |
| *** psilvad has quit IRC | 13:07 | |
| *** claudiub|4 is now known as claudiub|2 | 13:07 | |
| rcarrillocruz | cloudnull: nice, wasn't aware of that change, thx | 13:07 |
| Jeffrey4l | anyone check this? 'ubuntu-xenial-nv' suffix jobs are still running on stable/mitaka branch. https://review.openstack.org/356311 | 13:07 |
| *** registered78 has joined #openstack-infra | 13:07 | |
| *** zhurong has quit IRC | 13:07 | |
| *** psilvad has joined #openstack-infra | 13:08 | |
| cloudnull | sdague: ++ no doubt | 13:08 |
| *** mdrabe has joined #openstack-infra | 13:08 | |
| sdague | cloudnull: does xenial turn this off by default? | 13:08 |
| sdague | I'm curious why it just started to be an issue | 13:08 |
| cloudnull | but i will say that we found the isse because it was turned on (silver linning) :) | 13:08 |
| *** zhurong has joined #openstack-infra | 13:09 | |
| cloudnull | it seems most turn it off by default. but because DIB builds the images it wasn't part of the distro default. | 13:09 |
| cloudnull | xenial however has had improvements in handling temp addrs which is why it works there. | 13:10 |
| sdague | ok, so it's basically a dib issue in the fact that it bypasses all the distro defaults | 13:10 |
| *** matt-borland has joined #openstack-infra | 13:10 | |
| *** rbuzatu has joined #openstack-infra | 13:10 | |
| *** coolsvap is now known as _coolsvap_ | 13:11 | |
| cloudnull | it seems that way | 13:12 |
| cloudnull | we can likely make that same change to default dib | 13:12 |
| cloudnull | but i've only done it so far for the infra build | 13:12 |
| *** Julien-zte has quit IRC | 13:14 | |
| pabelanger | morning | 13:14 |
| *** rbuzatu has quit IRC | 13:15 | |
| mordred | Jeffrey4l: hrm | 13:15 |
| *** florianf has quit IRC | 13:16 | |
| mordred | cloudnull: yah - I think doing this in the base ubuntu element would be potentially a good thing | 13:16 |
| cloudnull | mordred: agreed. | 13:17 |
| mordred | Jeffrey4l: I'm not sure why that's happening, and I'm still on my first cup of coffee | 13:17 |
| *** inc0 has joined #openstack-infra | 13:17 | |
| *** sean-k-mooney has quit IRC | 13:18 | |
| Jeffrey4l | hmm OK. fyi, on the master branch, the ubunt-trusty-nv suffix job is not running( i.e. it works on master branch now) | 13:18 |
| *** baoli_ has quit IRC | 13:19 | |
| *** devkulkarni has quit IRC | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:20 | |
| *** baoli has joined #openstack-infra | 13:20 | |
| *** Julien-zte has joined #openstack-infra | 13:20 | |
| *** tqtran has joined #openstack-infra | 13:20 | |
| *** florianf has joined #openstack-infra | 13:20 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/diskimage-builder: Disabled IPv6 privacy extensions https://review.openstack.org/356456 | 13:23 |
| cloudnull | mordred: ^ | 13:23 |
| mordred | cloudnull: +1 from me! | 13:23 |
| *** dtantsur|bbl is now known as dtantsur | 13:24 | |
| rcarrillocruz | mordred: should we revert the osic trusty nodes change? | 13:24 |
| rcarrillocruz | cloudnull change has landed | 13:24 |
| tlbr | guys, could you please review/merge https://review.openstack.org/#/c/347047/ ? :( | 13:24 |
| *** _ari_ has quit IRC | 13:25 | |
| cloudnull | rcarrillocruz: do images need to be rebuilt ? | 13:25 |
| mordred | rcarrillocruz: we need new images | 13:25 |
| *** tqtran has quit IRC | 13:25 | |
| cloudnull | :'( | 13:25 |
| rcarrillocruz | yeah, one is ongoing | 13:25 |
| *** _ari_ has joined #openstack-infra | 13:25 | |
| rcarrillocruz | started 2h:48m ago | 13:25 |
| rcarrillocruz | not sure if the change merged before or after that | 13:26 |
| * rcarrillocruz goes check | 13:26 | |
| *** annegentle has joined #openstack-infra | 13:26 | |
| rcarrillocruz | bleh | 13:26 |
| rcarrillocruz | changed merge after the current periodic rebuild | 13:26 |
| *** aviau has quit IRC | 13:27 | |
| mordred | bleh indeed | 13:27 |
| *** aviau has joined #openstack-infra | 13:27 | |
| rcarrillocruz | i'll see how to rebuild manually | 13:27 |
| rcarrillocruz | syntax should be similar that snapshot images ( we didn't have dib on gozer at the time) | 13:27 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: add gnocchi stable/2.2 branch to jobs https://review.openstack.org/356459 | 13:27 |
| *** yamamoto has quit IRC | 13:29 | |
| rcarrillocruz | ah nice, there's no dib-image-update, image-update just does for dib | 13:29 |
| rcarrillocruz | good, i'll kick | 13:29 |
| pabelanger | you shouldn't need to | 13:29 |
| asselin | rcarrillocruz, nodepool image-build unless it changed | 13:30 |
| *** eandersson_ has joined #openstack-infra | 13:30 | |
| pabelanger | ubuntu-trusty DIBs haven't started yet, so it will be pulled in since DIB elements are already on nodepool.o.o | 13:30 |
| pabelanger | ubuntu-precise just finishing up now | 13:30 |
| rcarrillocruz | yup, thx | 13:30 |
| *** yamamoto has joined #openstack-infra | 13:30 | |
| *** pilgrimstack has quit IRC | 13:30 | |
| rcarrillocruz | pabelanger: should i kill it? | 13:30 |
| rcarrillocruz | i just kicked in an image-update manually myself on osic-cloud1 provider | 13:31 |
| pabelanger | no, I'll let all the image builds | 13:31 |
| pabelanger | gate isn't that bad right now | 13:31 |
| pabelanger | we have capacity I think | 13:31 |
| *** jianghuaw-m has joined #openstack-infra | 13:31 | |
| rcarrillocruz | so kill and upload on all providers | 13:31 |
| rcarrillocruz | ? | 13:32 |
| rcarrillocruz | for ubuntu-trusty that is | 13:32 |
| rcarrillocruz | ? | 13:32 |
| pabelanger | I'd say no | 13:32 |
| pabelanger | otherwise, you'll need to manually upload all other images | 13:32 |
| pabelanger | we're 1/2 way into image builds now | 13:32 |
| pabelanger | uploads should start in another 3 hours | 13:32 |
| *** javeriak has quit IRC | 13:33 | |
| rcarrillocruz | how long does it usually take | 13:33 |
| rcarrillocruz | an image build | 13:33 |
| rcarrillocruz | 4h | 13:33 |
| rcarrillocruz | ? | 13:33 |
| pabelanger | each build is about a 1 hour these days | 13:33 |
| *** woodster_ has joined #openstack-infra | 13:34 | |
| pabelanger | we've been averaging about 12hour cycles for build / uploads across all providers | 13:34 |
| rcarrillocruz | cos i did 'nodepool dib-image-list' and i see the ubuntu-trusty has been ongoing for 3h | 13:34 |
| pabelanger | yup | 13:35 |
| pabelanger | image builds are serial | 13:35 |
| pabelanger | only 1 at a time | 13:35 |
| rcarrillocruz | ah right | 13:35 |
| rcarrillocruz | gotcha | 13:35 |
| pabelanger | so, once ubuntu-trusty is build, you should be able to manually upload it to osic-cloud1 | 13:35 |
| pabelanger | currently nodepool-builder waits for all image builds before uploading | 13:36 |
| pabelanger | I' | 13:36 |
| openstackgerrit | Emily Hugenbruch proposed openstack-infra/irc-meetings: Create a new meeting for WOS-mentoring https://review.openstack.org/356467 | 13:36 |
| rcarrillocruz | the cron job doesn't upload then ? where are the images kept? | 13:36 |
| *** caowei has joined #openstack-infra | 13:36 | |
| pabelanger | I've been meaning to fix that, but need to dive into python threads | 13:36 |
| *** annegentle has quit IRC | 13:36 | |
| pabelanger | no, it does | 13:36 |
| *** pilgrimstack has joined #openstack-infra | 13:36 | |
| pabelanger | only after all images have been built | 13:36 |
| rcarrillocruz | ok | 13:37 |
| pabelanger | /opt/nodepool_dib is the images | 13:37 |
| pabelanger | nodepool-builder should start uploads as image builds finish | 13:37 |
| pabelanger | but needs patch | 13:37 |
| Shrews | pabelanger: new builder should take care of that | 13:37 |
| *** edmondsw has joined #openstack-infra | 13:37 | |
| mordred | \o/ | 13:37 |
| *** mtanino has joined #openstack-infra | 13:37 | |
| Shrews | when i finish it, that is | 13:37 |
| pabelanger | Shrews: yay, I was hoping that was the case | 13:38 |
| rcarrillocruz | ++ | 13:38 |
| *** dizquierdo has joined #openstack-infra | 13:38 | |
| Shrews | pabelanger: referring to https://review.openstack.org/#/c/356079/1/doc/source/devguide.rst, when scheduler gets notified of build done, upload can start | 13:39 |
| *** jcoufal has joined #openstack-infra | 13:39 | |
| *** tongli has joined #openstack-infra | 13:39 | |
| Shrews | in theory at least | 13:39 |
| pabelanger | rcarrillocruz: mordred: I actually think we should consider using another flavor in rackspace for nodepool-builder more friendly to disk IO. Since that is our bottleneck right now | 13:40 |
| pabelanger | We use performance1-8 today | 13:40 |
| *** yaume has joined #openstack-infra | 13:41 | |
| rcarrillocruz | cinerama: helo, when you get around it would be great if you could review https://review.openstack.org/#/c/355778/ . Currently, the config on baremetal00 is flip-flopping between baremetal00 node config and openstack_project::server. As soon as the install runs by puppet, the play breaks fqdn resolution, then puppet apply doesn't match 'baremetal00.ic.vanilla.openstack.org' and just applies manifests | 13:41 |
| rcarrillocruz | for openstack_project::server because facter fqdn returns non-fqdn :/ | 13:41 |
| rcarrillocruz | pabelanger: sounds sensible to me, we should make some testing and get numbers on the potential gain | 13:41 |
| pabelanger | we could actually launch a dedicated nodepool-builder.o.o server too, do testing, and once working bring it online | 13:42 |
| rcarrillocruz | pabelanger: ^ this is why unbound is enabled on baremetal00, despite being explicitly disabled on the site.pp node declaration | 13:42 |
| rcarrillocruz | yeah | 13:43 |
| rcarrillocruz | which reminds me, i need to sync up with ianw , i think he wanted to run the gerrit-dev machine launch | 13:43 |
| *** gildub has quit IRC | 13:45 | |
| *** Jaison has quit IRC | 13:46 | |
| openstackgerrit | Merged openstack-infra/storyboard: Fix docs so it is easy to understand https://review.openstack.org/355886 | 13:46 |
| openstackgerrit | Babu Shanmugam proposed openstack-infra/project-config: Move networking-ovn grenade CI job from experimental to check https://review.openstack.org/356474 | 13:47 |
| *** xarses_ has quit IRC | 13:47 | |
| *** xarses_ has joined #openstack-infra | 13:48 | |
| *** esikachev has quit IRC | 13:50 | |
| *** sdake has quit IRC | 13:50 | |
| mordred | pabelanger: I think we should get the first bit of Shrews new builder stuff landed before we start scaling out builders | 13:50 |
| mordred | pabelanger: just so that our transition surface is smaller | 13:50 |
| *** xyang1 has quit IRC | 13:51 | |
| mordred | pabelanger: but then I totally agree with you | 13:51 |
| mordred | (I mean, I totally agree with you know ...) | 13:51 |
| *** xyang1 has joined #openstack-infra | 13:51 | |
| mordred | s/know/now/ | 13:51 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar https://review.openstack.org/355554 | 13:52 |
| *** camunoz has joined #openstack-infra | 13:52 | |
| *** jianghuaw-m has quit IRC | 13:54 | |
| openstackgerrit | Zsolt Dudás proposed openstack-infra/jenkins-job-builder: Replaces logging with oslo.log https://review.openstack.org/354850 | 13:55 |
| pabelanger | mordred: Ya, we need to update our manifests first. Starting to poke at that now | 13:57 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 13:57 |
| *** pt_15 has joined #openstack-infra | 13:58 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 13:58 |
| *** xyang1 has quit IRC | 13:58 | |
| *** apetrich has quit IRC | 13:58 | |
| tlbr | mordred, pabelanger, fungi could you please review https://review.openstack.org/#/c/347047/ ? | 13:59 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Provide unit test coverage for process_results https://review.openstack.org/355385 | 13:59 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting https://review.openstack.org/355393 | 13:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: include ::diskimage_builder in builder.pp too https://review.openstack.org/356484 | 13:59 |
| mordred | oh wow | 13:59 |
| mordred | tlbr: that is a patch! ) | 13:59 |
| mordred | :) | 13:59 |
| pabelanger | no doubt | 13:59 |
| tlbr | yep:) | 14:00 |
| *** sdake_ has joined #openstack-infra | 14:01 | |
| *** ociuhandu has quit IRC | 14:01 | |
| pabelanger | +3 | 14:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 14:02 |
| pabelanger | oh, that reminds me | 14:03 |
| tlbr | thank you! | 14:03 |
| pabelanger | mordred: care to review: https://review.openstack.org/#/c/354818/ | 14:03 |
| pabelanger | I have some time today to monitor it | 14:04 |
| openstackgerrit | Aleksey Zvyagintsev proposed openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool. https://review.openstack.org/353861 | 14:04 |
| fungi | cloudnull: sdague: confirmed at least for fixing our traceroute6 behavior, that the held ubuntu-trusty node i have in osic suddenly began working after i did `sudo sysctl -w net.ipv6.conf.all.use_tempaddr=0 ; sudo sysctl -w net.ipv6.conf.default.use_tempaddr=0` | 14:04 |
| fungi | good find | 14:04 |
| *** amotoki has quit IRC | 14:05 | |
| *** armax has quit IRC | 14:05 | |
| *** esberglu has joined #openstack-infra | 14:05 | |
| cloudnull | im hoping that make the osic gate +/- happy | 14:05 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use separated SSL endpoint environment file https://review.openstack.org/356488 | 14:05 |
| sdague | fungi: ok, once that's in the image we can revert the thing rcarrillocruz did to disable trusty on osic | 14:06 |
| *** xarses_ has quit IRC | 14:06 | |
| cloudnull | ++ | 14:06 |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| zigo | pabelanger: mordred: Thanks for the +3 ! :) | 14:07 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Fix bridge creation when no vlan is involved https://review.openstack.org/355845 | 14:07 |
| zigo | pabelanger: mordred: Now, we're really starting ... | 14:07 |
| *** Rockyg has joined #openstack-infra | 14:07 | |
| *** pradk has joined #openstack-infra | 14:08 | |
| rcarrillocruz | yolanda: your change passed ^ | 14:08 |
| *** SamYaple has quit IRC | 14:09 | |
| *** SamYaple has joined #openstack-infra | 14:09 | |
| mordred | zigo: \o/ | 14:09 |
| yolanda | woot! | 14:09 |
| fungi | tlbr: is there a corresponding governance change for 347047? can you confirm that at least all the repos which were added to governance in https://review.openstack.org/185187 are covered by it? | 14:09 |
| zigo | mordred: FYI, on purpose, we didn't add a .gitreview. This will be the first commits on all repos, which will make them build. | 14:09 |
| mordred | zigo: I think that's a great plan | 14:10 |
| *** tonytan4ever has joined #openstack-infra | 14:10 | |
| *** SamYaple has quit IRC | 14:11 | |
| pabelanger | zigo: mordred: Ya, once we mirror package sources, I think you'll be in good shape | 14:11 |
| *** SamYaple has joined #openstack-infra | 14:11 | |
| vgridnev | another try: hello folks, could you please review https://review.openstack.org/#/c/354700/ ? thanks in advance | 14:11 |
| zigo | Oh, joy, telnet with an ipv6 link! :) | 14:12 |
| *** florianf has quit IRC | 14:12 | |
| pabelanger | zigo: mordred: can you do a new build on deb-openstack-pkg-tools? I want to see if gpp will sign the Release file | 14:12 |
| * zigo loves his dual-stack ISP... | 14:12 | |
| *** armax has joined #openstack-infra | 14:12 | |
| *** amotoki has joined #openstack-infra | 14:12 | |
| *** adrian_otto has joined #openstack-infra | 14:12 | |
| *** ociuhandu has joined #openstack-infra | 14:12 | |
| zigo | pabelanger: I'll be doing that as soon as my @(#$@ deb-auto-backports script works. | 14:12 |
| zigo | I should be close to it. | 14:13 |
| * cloudnull wishes my ISP would restore dual stack :\ | 14:13 | |
| zigo | It's "only" the 16th iteration of my patch... :/ | 14:13 |
| *** SamYaple has quit IRC | 14:13 | |
| *** dizquierdo has quit IRC | 14:13 | |
| *** SamYaple has joined #openstack-infra | 14:13 | |
| *** ihrachys has joined #openstack-infra | 14:13 | |
| *** sarob has joined #openstack-infra | 14:13 | |
| zigo | Arg, failure ... :/ | 14:13 |
| *** spzala_ has quit IRC | 14:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Remove nodepool-builder from nodepool::init https://review.openstack.org/356498 | 14:14 |
| *** spzala has joined #openstack-infra | 14:14 | |
| *** florianf has joined #openstack-infra | 14:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackci: Use nodepool::builder for nodepool https://review.openstack.org/356503 | 14:16 |
| *** SamYaple has quit IRC | 14:17 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 14:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Clarify code in nodepool/cmd/builder.py https://review.openstack.org/356504 | 14:17 |
| *** hongbin has joined #openstack-infra | 14:17 | |
| *** spzala has quit IRC | 14:17 | |
| *** adrian_otto has quit IRC | 14:17 | |
| *** spzala has joined #openstack-infra | 14:17 | |
| *** SamYaple has joined #openstack-infra | 14:17 | |
| rcarrillocruz | doh | 14:18 |
| rcarrillocruz | another conversion string to dict thing on bifrost, patch incoming | 14:18 |
| *** sarob has quit IRC | 14:18 | |
| fungi | zigo: do you have direct telnet:// url integration in your browser? i'm curious if that is appropriate formatting or whether the address needs to be bracketed in [] to differentiate it from the port number/separator | 14:18 |
| *** zz_dimtruck is now known as dimtruck | 14:19 | |
| fungi | i just wind up pasting the url and editing it into parameters for netcat anyway | 14:19 |
| openstackgerrit | Merged openstack-infra/shade: Lay the groundwork for per-resource cache https://review.openstack.org/339322 | 14:19 |
| fungi | but would be good to make sure that's actually a legitimate telnet:// url | 14:19 |
| openstackgerrit | Merged openstack-infra/shade: Create and return per-resource caches https://review.openstack.org/339323 | 14:19 |
| openstackgerrit | Merged openstack-infra/shade: Move list_ports to using dogpile.cache https://review.openstack.org/339324 | 14:19 |
| openstackgerrit | Merged openstack-infra/shade: Add tests to show IP inference in missed conditions https://review.openstack.org/355183 | 14:20 |
| openstackgerrit | Merged openstack-infra/shade: Base auto_ip on interface_ip not public_v4 https://review.openstack.org/355048 | 14:20 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: StoryBoard 'about' page tweak https://review.openstack.org/354570 | 14:20 |
| *** hockeynut has joined #openstack-infra | 14:20 | |
| *** fguillot has joined #openstack-infra | 14:21 | |
| *** ihrachys has quit IRC | 14:21 | |
| *** ihrachys has joined #openstack-infra | 14:21 | |
| *** caowei has quit IRC | 14:21 | |
| rcarrillocruz | mordred: https://review.openstack.org/#/c/356506/1 when you get a sec | 14:21 |
| rcarrillocruz | TheJulia: ^ | 14:22 |
| fungi | oh, also, for anyone on debuntu with ipv6 trying to figure out why their netcat doesn't support it, you want netcat-openbsd not netcat-traditional | 14:22 |
| openstackgerrit | Merged openstack-infra/shade: Validate config vs reality better than length of list https://review.openstack.org/355248 | 14:22 |
| openstackgerrit | Merged openstack-infra/shade: Honor default_interface OCC setting in create_server https://review.openstack.org/355250 | 14:22 |
| *** valderrv has joined #openstack-infra | 14:22 | |
| Shrews | wooo, openstackgerrit casting shade | 14:22 |
| *** caowei has joined #openstack-infra | 14:22 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add symlink to elastic-recheck uncategorized.html https://review.openstack.org/356508 | 14:22 |
| pleia2 | mtreinish: ^^ | 14:23 |
| tlbr | fungi, may be you mean https://review.openstack.org/#/c/348077/ ? yes, i can confirm that it is in governance | 14:23 |
| mtreinish | pleia2: cool, thanks | 14:23 |
| *** ihrachys has quit IRC | 14:23 | |
| tlbr | i'll add mistral-dashboard once it is packaged. Just need a bit more time to do that | 14:24 |
| *** claudiub has joined #openstack-infra | 14:24 | |
| *** claudiub|4 has joined #openstack-infra | 14:24 | |
| *** mdrabe has quit IRC | 14:24 | |
| *** ihrachys has joined #openstack-infra | 14:24 | |
| *** claudiu0 has joined #openstack-infra | 14:25 | |
| *** claudiu7 has joined #openstack-infra | 14:25 | |
| zigo | fungi: https://addons.mozilla.org/en-US/firefox/addon/ipprotocols/ | 14:25 |
| *** claudiub|2 has quit IRC | 14:25 | |
| *** claudiub|3 has quit IRC | 14:25 | |
| cloudnull | i dont always netcat, but when I do I netcat-openbsd | 14:25 |
| cloudnull | :p | 14:25 |
| beagles | I was keeping a weather eye on this channel, but did I see you guys worked out the network thing on osic? | 14:26 |
| beagles | cloudnull, ^ | 14:26 |
| rcarrillocruz | so folks, bifrost bakes a deploy ssh key on the config drives, prior to provisioning those servers. In Gozer , I had a special purpose user, which I sudo su everytime to do bifrost deployments and used its /home/<user>/.ssh/id_rsa.pub file for the aforementioned deployment key | 14:27 |
| rcarrillocruz | since i think it's a bit of a pain given all root users we are | 14:27 |
| *** yamamoto has quit IRC | 14:27 | |
| rcarrillocruz | it would be good if we could just run ansible bifrost plays as our own users | 14:27 |
| fungi | beagles: yes, cloudnull figured out it was v6 privacy extensions mucking with routing | 14:27 |
| rcarrillocruz | i'm thinking on putting a deploy key on /etc/bifrost | 14:27 |
| beagles | wow | 14:27 |
| cloudnull | beagles: ohai, I think so. we made this change in the infra image and later in dib https://review.openstack.org/#/q/branch:master+topic:bug/1068756 | 14:27 |
| fungi | beagles: the images building today will have it disabled | 14:27 |
| rcarrillocruz | and put that path on the bifrost global vars | 14:28 |
| rcarrillocruz | objections to using that location? | 14:28 |
| *** yamahata has joined #openstack-infra | 14:28 | |
| beagles | fungi, cloudnull, that is impressive! | 14:28 |
| beagles | fungi, cloudnull, nice bit of work! | 14:28 |
| rcarrillocruz | i.e. we would have something like /etc/bifrost/id_rsa_deploy.pub and /etc/bifrost/id_rsa_deploy | 14:28 |
| *** claudiub has quit IRC | 14:28 | |
| *** fguillot has quit IRC | 14:28 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/elastic-recheck: Fix active page menu coloring https://review.openstack.org/356509 | 14:28 |
| *** claudiub|4 has quit IRC | 14:28 | |
| cloudnull | beagles: thanks -- fungi: did the heavy lifting, I'm just a button masher. | 14:29 |
| *** yamamoto has joined #openstack-infra | 14:29 | |
| pleia2 | mtreinish: ^^ also, that was bugging me ;) | 14:29 |
| fungi | beagles: all credit to cloudnull. i saw the extra non-eui64 global addresses bound to eth0 and didn't give it a second thought | 14:29 |
| beagles | lol | 14:29 |
| *** kushal has joined #openstack-infra | 14:29 | |
| *** xarses_ has joined #openstack-infra | 14:30 | |
| *** itisha has joined #openstack-infra | 14:30 | |
| mtreinish | pleia2: +A | 14:30 |
| beagles | pretty wild... I've got to amp up my ipv6-fu | 14:30 |
| rcarrillocruz | pabelanger: ^ | 14:30 |
| *** njohnston has left #openstack-infra | 14:30 | |
| *** claudiu0 has quit IRC | 14:31 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Update networking-ovn grafana dashboard https://review.openstack.org/356511 | 14:31 |
| *** claudiu7 has quit IRC | 14:32 | |
| *** mdrabe has joined #openstack-infra | 14:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new/update existing projects https://review.openstack.org/347047 | 14:33 |
| beagles | we've no real support for in my part of the world so I only get to see it in synthetic environments :( | 14:33 |
| beagles | anyways, awesome stuff! | 14:33 |
| rcarrillocruz | ++ | 14:33 |
| *** yamamoto has quit IRC | 14:34 | |
| *** sarob has joined #openstack-infra | 14:34 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 14:35 |
| *** edtubill has joined #openstack-infra | 14:35 | |
| cloudnull | once we get some of these issues sorted out, I'd like to once again up the quota to something like 1024 instances and really see infra attack the OSIC cloud. | 14:36 |
| mordred | cloudnull: we would also like to see that | 14:36 |
| cloudnull | :) | 14:36 |
| cloudnull | no sleep til it's broken! | 14:37 |
| mordred | cloudnull: we've broken clouds before - I'm sure we'll break clouds again | 14:37 |
| * cloudnull all about breaking clouds | 14:37 | |
| mordred | it's the only way to make them better | 14:37 |
| cloudnull | +1 | 14:37 |
| cloudnull | its simple to keep a cloud stable when nobody is using it | 14:38 |
| * beagles feels a new deployment project called "salter" coming on | 14:38 | |
| *** pgadiya has quit IRC | 14:38 | |
| fungi | cloudnull: also apparently inc0 and mrhillsman want to do something similar with us providing a test load for various deployment attempts in osic | 14:38 |
| beagles | actually that's "seeding" I suppose | 14:38 |
| cloudnull | ++ yup, that'd be great too | 14:38 |
| inc0 | yeah, that's exactly right cloudnull, we want us to be tested hard;) | 14:39 |
| fungi | cloudnull: but we'll just selfishly run on as much quota as you can provide. especially this time of the cycle | 14:39 |
| inc0 | and it's my understanding that infra excels at this;) | 14:39 |
| fungi | we're merely facilitators. it's our community who excel at loading up these systems with code to be tested | 14:39 |
| inc0 | btw fungi, me and mrhillsman are pretty damn close to get it up | 14:40 |
| jeblair | i have personally broken two public clouds | 14:40 |
| mrhillsman | can we use ipv6 for the local-mirror? | 14:40 |
| fungi | jeblair: i'm sure that's not a record. you should try harder | 14:40 |
| inc0 | good, break ours as well, so we can find the weaknesses and make sure projects knows about them | 14:40 |
| *** annegentle has joined #openstack-infra | 14:40 | |
| jeblair | fungi: it's the team best that matters most :) | 14:41 |
| *** sarob has quit IRC | 14:41 | |
| fungi | mrhillsman: yes and no. our job nodes will be connecting to the mirror via ipv6, but the mirror still needs working ipv4 because it uses afs for the data cache | 14:41 |
| inc0 | btw fungi that's our ultimate goal - you load test us so we can provide feedback for project teams | 14:41 |
| mrhillsman | ok cool | 14:41 |
| mrhillsman | we should be ready to rock and roll today then | 14:41 |
| *** sarob has joined #openstack-infra | 14:41 | |
| mrhillsman | that was the last thing i believe we needed to know | 14:41 |
| *** ekhugen has joined #openstack-infra | 14:42 | |
| *** Hal has quit IRC | 14:42 | |
| zigo | pabelanger: https://review.openstack.org/#/c/354632/ just got a +2, 2 new packages will be comming. | 14:43 |
| pabelanger | zigo: okay, watching | 14:43 |
| pabelanger | fungi: Do we need ipv4 for API endpoints or does ipv6 work? | 14:44 |
| pabelanger | jeblair: we need to get you a t-shirt that says that! | 14:44 |
| fungi | pabelanger: aiui, shade _should_ be fine with ipv6 api endpoints, as long as the keystone catalog can do that correctly | 14:44 |
| pabelanger | fungi: right, that's what I would expect too | 14:45 |
| *** registered78 has quit IRC | 14:45 | |
| pabelanger | mrhillsman: So, ipv4 works, ipv6 means some testing for endpoints. | 14:45 |
| fungi | pabelanger: so it's less about whether _we_ can use ipv6 openstack apis, but more about whether openstack ipv6 api endpoints work at all | 14:45 |
| mrhillsman | got it | 14:45 |
| pabelanger | fungi: ++ | 14:45 |
| mrhillsman | yeah, i do not think openstack ipv6 api endpoints are working at this time | 14:46 |
| mrhillsman | something for us to help with going forward :) | 14:46 |
| fungi | any client-side bugs can probably get fixed very quickly. service-side bugs are something that will take more time | 14:46 |
| mordred | I think shade actually will never see anything related ot v4/v6 endpoints | 14:46 |
| mordred | that's all handled by keystoneauth | 14:46 |
| fungi | mordred: ahh, right that then | 14:46 |
| fungi | so even lower in the dependency stack | 14:47 |
| mordred | notmorgan: any idea if keystoneauth has done any validation on ipv6 endpoints? | 14:47 |
| *** ihrachys has quit IRC | 14:47 | |
| zigo | fungi: The "BBSFox" browser plugin seems to handle telnet:// correctly as well. | 14:48 |
| fungi | zigo: thanks for the tips! i'll give those a try in a bit for testing purposes | 14:48 |
| *** ihrachys has joined #openstack-infra | 14:49 | |
| *** sarob has quit IRC | 14:49 | |
| *** dizquierdo has joined #openstack-infra | 14:49 | |
| *** nmagnezi has quit IRC | 14:49 | |
| *** sarob has joined #openstack-infra | 14:49 | |
| *** valderrv has quit IRC | 14:49 | |
| *** csomerville has quit IRC | 14:49 | |
| *** csomerville has joined #openstack-infra | 14:50 | |
| clarkb | I am pretty sure the puppet team tests everything with ipv6 | 14:50 |
| *** ihrachys has quit IRC | 14:50 | |
| clarkb | so it should work but they would have better info | 14:51 |
| clarkb | EmilienM: ^ | 14:51 |
| *** ihrachys has joined #openstack-infra | 14:51 | |
| EmilienM | hi | 14:51 |
| EmilienM | yes, our CI deploy all OpenStack in ipv6 on centos7 (not on ubuntu xenial, last time we tried it didn't work) | 14:52 |
| *** derekh has quit IRC | 14:52 | |
| EmilienM | it's documented here: https://github.com/openstack/puppet-openstack-integration#description | 14:52 |
| mordred | neat! | 14:52 |
| EmilienM | let me know if someone needs help to deploy that | 14:54 |
| *** infra-red has quit IRC | 14:55 | |
| pabelanger | fungi: have a moment to review 354818? I have some bandwidth today to manage that | 14:56 |
| *** dizquierdo has quit IRC | 14:57 | |
| fungi | pabelanger: we're doing okay on available space/quota for our mirror volumes to support that? | 14:57 |
| fungi | pabelanger: also, it probably wouldn't hurt to check over our cache sizes and see if we need to bump those | 14:57 |
| *** valderrv has joined #openstack-infra | 14:58 | |
| *** rvba has quit IRC | 14:59 | |
| *** zhurong has quit IRC | 14:59 | |
| notmorgan | mordred: uhm | 14:59 |
| notmorgan | mordred: what kind of validation are you thinking about? | 14:59 |
| *** elo has quit IRC | 14:59 | |
| *** martinkopec_ has joined #openstack-infra | 15:00 | |
| pabelanger | fungi: I'll have to bump the mirror.debian up to match mirror.ubuntu. We are aleady at 70% capacity there | 15:00 |
| *** martinkopec has quit IRC | 15:00 | |
| pabelanger | mirror.ubuntu is 50% right now | 15:00 |
| mordred | notmorgan: that it works ... although it seems that puppet-openstack has tested ipv6-only ... so it should | 15:01 |
| *** _coolsvap_ is now known as coolsvap | 15:01 | |
| *** pilgrimstack has quit IRC | 15:01 | |
| amrith | is it just me or does the CI queue look a bit long this morning? | 15:01 |
| pabelanger | jeblair: fungi: as for our afs01.dfw.o.o, our volume is at 70% capacity. We should consider adding another volume | 15:02 |
| pabelanger | 615GB available | 15:02 |
| *** Hal has joined #openstack-infra | 15:02 | |
| fungi | amrith: not as bad as yesterday, but when the americas wake up and start pushing/approving changes the queues start to grow quickly | 15:03 |
| amrith | those americans ... | 15:03 |
| fungi | yeah, so many of them | 15:03 |
| *** openstackgerrit has quit IRC | 15:03 | |
| *** openstackgerrit has joined #openstack-infra | 15:04 | |
| *** vikrant has joined #openstack-infra | 15:04 | |
| *** vikrant has quit IRC | 15:04 | |
| pabelanger | amrith: I guess you didn't see last week. We were pushing 600 in check queue at one point | 15:04 |
| notmorgan | mordred: uhm, i don't think we test ipv6 explicitly, we probably have a check | 15:04 |
| notmorgan | in our tests that is | 15:04 |
| amrith | ok folks, would all of you please stop sending patches. I have one that I'd like tested :) | 15:05 |
| *** sarob has quit IRC | 15:07 | |
| amrith | let's see if that works :) | 15:07 |
| *** sarob has joined #openstack-infra | 15:07 | |
| *** vgridnev has quit IRC | 15:07 | |
| *** raunak has joined #openstack-infra | 15:07 | |
| *** xyang1 has joined #openstack-infra | 15:07 | |
| *** ihrachys has quit IRC | 15:08 | |
| pabelanger | jeblair: fungi: I think I missed the final discussions from yesterday about urandom for ubuntu-xenial. I _think_ we are going try installing rng-tools? | 15:09 |
| *** ihrachys has joined #openstack-infra | 15:09 | |
| jeblair | pabelanger: i have an alternate proposal | 15:09 |
| fungi | pabelanger: jeblair tested it as well and got some more data | 15:09 |
| fungi | i'll let him elaborate ;) | 15:09 |
| pabelanger | great | 15:09 |
| *** martinkopec_ is now known as martinkopec | 15:09 | |
| *** martinkopec is now known as martinkopec_ | 15:09 | |
| jeblair | pabelanger: i would like to write an entropy pump that we run at startup to feed data from haveged to the kernel so that urandom initializes quickly | 15:09 |
| jeblair | pabelanger: i think that will be easy to do and won't require any other software | 15:10 |
| *** amotoki has quit IRC | 15:10 | |
| pabelanger | okay, perfect | 15:10 |
| *** vgridnev has joined #openstack-infra | 15:10 | |
| jeblair | pabelanger: if i work on writing the program, can you work on getting it to run very early in the init process? | 15:11 |
| *** dprince has quit IRC | 15:11 | |
| *** ihrachys has quit IRC | 15:11 | |
| pabelanger | jeblair: sure | 15:11 |
| *** rbuzatu has joined #openstack-infra | 15:12 | |
| *** ihrachys has joined #openstack-infra | 15:12 | |
| clarkb | huh haveged doesnt already function that way? I guess I dont understand how haveged's entropy ends up being useable by userspace programs | 15:13 |
| jeblair | pabelanger: i should have that in a few hours | 15:13 |
| *** cody-somerville has joined #openstack-infra | 15:14 | |
| *** cody-somerville has joined #openstack-infra | 15:14 | |
| jeblair | clarkb: you can find out either by reading backscroll from me yesterday, or wait until i finish this program which will have an explanatory essay at the top. :) | 15:14 |
| pabelanger | jeblair: okay, I'll have systemd ready for then | 15:14 |
| fungi | clarkb: it doesn't quite push enough in on its first pulse to trigger crediting entropy into the nonblocking pool | 15:14 |
| fungi | but yeah, i'm sure jeblair's writeup will include plenty of useful citations | 15:15 |
| fungi | ultimately, this is _probably_ something that could be solved in haveged | 15:15 |
| *** csomerville has quit IRC | 15:16 | |
| *** ihrachys has quit IRC | 15:16 | |
| *** ihrachys has joined #openstack-infra | 15:16 | |
| fungi | not sure if they know this is an issue in newer kernels and have chosen not to address it, or what | 15:16 |
| *** rbuzatu has quit IRC | 15:16 | |
| zigo | pabelanger: http://logs.openstack.org/96/96d617f2d515d4a78b15af0535bb17ad23c9fa30/post/reprepro-sign-jessie-backports/e260d75/console.html <--- Signing failed. | 15:17 |
| *** vgridnev has quit IRC | 15:17 | |
| *** oanson has quit IRC | 15:17 | |
| zigo | pabelanger: Where's your script? | 15:17 |
| *** Julien-zte has quit IRC | 15:18 | |
| pabelanger | Blarg | 15:19 |
| pabelanger | I see the issue | 15:19 |
| *** vgridnev has joined #openstack-infra | 15:19 | |
| pabelanger | zigo: in project-config, I can write a patch now | 15:19 |
| zigo | Ok. | 15:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix typo with AFS_DIR https://review.openstack.org/356546 | 15:20 |
| pabelanger | zigo: ^ | 15:20 |
| *** raunak has quit IRC | 15:21 | |
| zigo | Oh ! | 15:21 |
| pabelanger | mordred: fungi: might helping review ^ | 15:21 |
| pabelanger | zigo: so close | 15:22 |
| zigo | :) | 15:22 |
| *** kzaitsev_mb has quit IRC | 15:23 | |
| *** yaume has quit IRC | 15:24 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:24 | |
| *** vgridnev has quit IRC | 15:24 | |
| *** florianf has quit IRC | 15:25 | |
| *** mdrabe has quit IRC | 15:25 | |
| *** vgridnev has joined #openstack-infra | 15:25 | |
| *** vgridnev has quit IRC | 15:25 | |
| *** mdrabe has joined #openstack-infra | 15:25 | |
| *** ihrachys has quit IRC | 15:26 | |
| *** ihrachys has joined #openstack-infra | 15:26 | |
| *** markusry has joined #openstack-infra | 15:27 | |
| *** thiagop has joined #openstack-infra | 15:29 | |
| *** caowei has quit IRC | 15:29 | |
| *** ihrachys has quit IRC | 15:30 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 15:30 |
| *** vinaypotluri has joined #openstack-infra | 15:30 | |
| *** ihrachys has joined #openstack-infra | 15:30 | |
| *** florianf has joined #openstack-infra | 15:32 | |
| betherly | hi all! just wondering if i have the releasenotes for ironic-ui set up correctly as they are not building at http://docs.openstack.org/releasenotes/ironic-ui | 15:32 |
| *** yamamoto has joined #openstack-infra | 15:32 | |
| *** ihrachys has quit IRC | 15:32 | |
| *** Swami has joined #openstack-infra | 15:34 | |
| *** Sukhdev has joined #openstack-infra | 15:35 | |
| *** yamamoto has quit IRC | 15:37 | |
| *** rbuzatu has joined #openstack-infra | 15:39 | |
| pabelanger | fungi: did you see the ML thread [OpenStack-Infra] Importing code with history (sahara-tests)? I am pretty sure we can do it, just wanted to confirm before offering up help | 15:40 |
| karthikp_ | Hi all, | 15:40 |
| karthikp_ | Need your help to review one of our patches required for enabling multinode grenade testing in Cinder: https://review.openstack.org/#/c/355678/ | 15:41 |
| fungi | pabelanger: i'm still catching up on the ml from overnight. haven't gotten that far yet | 15:42 |
| fungi | thanks for the heads up | 15:42 |
| *** ihrachys has joined #openstack-infra | 15:42 | |
| anteaya | karthikp_: best to get some cinder folks to review that patch and state they are aware and want this test | 15:43 |
| karthikp_ | anteaya: we already some patches merged for this | 15:43 |
| *** sarob has quit IRC | 15:44 | |
| anteaya | karthikp_: then you need grenade reviewers to review it, they spend time in #openstack-qa | 15:44 |
| karthikp_ | where other folks have showed interest | 15:44 |
| *** matrohon has quit IRC | 15:46 | |
| *** ihrachys has quit IRC | 15:48 | |
| *** abregman has quit IRC | 15:49 | |
| *** ihrachys has joined #openstack-infra | 15:49 | |
| *** ihrachys has quit IRC | 15:51 | |
| *** jaosorior has quit IRC | 15:51 | |
| *** martinkopec has joined #openstack-infra | 15:52 | |
| *** ihrachys has joined #openstack-infra | 15:53 | |
| *** mhickey has quit IRC | 15:53 | |
| *** martinkopec_ has quit IRC | 15:53 | |
| *** amotoki has joined #openstack-infra | 15:56 | |
| *** amitgandhinz has joined #openstack-infra | 15:57 | |
| *** rcernin has quit IRC | 15:58 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add scheduling thread to nodepool builder https://review.openstack.org/356079 | 15:59 |
| *** bodepd has quit IRC | 15:59 | |
| Shrews | jeblair: that just changes the commit message ^^^ | 15:59 |
| *** amitgandhinz has quit IRC | 15:59 | |
| *** dtantsur is now known as dtantsur|brb | 16:00 | |
| jeblair | Shrews: ++ | 16:00 |
| *** amitgandhinz has joined #openstack-infra | 16:00 | |
| *** sdake_ is now known as sdake | 16:00 | |
| *** sdake is now known as sdake_ | 16:00 | |
| *** thorst_ has joined #openstack-infra | 16:01 | |
| *** markusry has quit IRC | 16:02 | |
| *** weshay is now known as weshay_bbiab | 16:03 | |
| *** andymaier has quit IRC | 16:03 | |
| *** wewe0901 has joined #openstack-infra | 16:04 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/project-config: Few changed to the kolla-kubernetes job https://review.openstack.org/355199 | 16:04 |
| *** dmellado is now known as dmellado|off | 16:05 | |
| *** matthewbodkin has quit IRC | 16:07 | |
| *** pilgrimstack has joined #openstack-infra | 16:08 | |
| *** Apoorva has joined #openstack-infra | 16:08 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make task notes read-only by default https://review.openstack.org/325294 | 16:09 |
| pabelanger | async task produced unparseable results: http://logs.openstack.org/18/354818/1/gate/gate-infra-puppet-apply-ubuntu-precise/ed69460/_zuul_ansible/ansible_log.txt | 16:10 |
| *** Cormite has joined #openstack-infra | 16:10 | |
| pabelanger | didn't we patch that already? | 16:10 |
| mordred | jeblair, Shrews: I am reviewing that patch - but it's big, so it might take a sec | 16:10 |
| *** puiterwijk has quit IRC | 16:11 | |
| Shrews | mordred: if you feel that patch is big, then it was the right choice to go ahead and try to merge it. :) the stuff depending on it is much larger i'm afraid | 16:11 |
| *** ihrachys has quit IRC | 16:12 | |
| *** Hal has quit IRC | 16:13 | |
| *** Hal has joined #openstack-infra | 16:13 | |
| mordred | Shrews: oh good | 16:13 |
| *** Cormite has quit IRC | 16:15 | |
| *** Cormite has joined #openstack-infra | 16:15 | |
| zaro | morning | 16:15 |
| *** _nadya_ has quit IRC | 16:16 | |
| betherly | fungi: sorry to ping you on this! wondering if i have the releasenotes for ironic-ui set up correctly as they are not building at http://docs.openstack.org/releasenotes/ironic-ui | 16:17 |
| anteaya | morning zaro | 16:17 |
| anteaya | pabelanger: yes, I had thought that was fixed | 16:17 |
| *** vhosakot has joined #openstack-infra | 16:18 | |
| anteaya | pabelanger: that is from today :( | 16:18 |
| fungi | betherly: release notes updates for merged changes (not tagged releases) get processed in the post pipeline, which gets pretty backed up | 16:19 |
| pabelanger | anteaya: Thanks, I'll check ansible to see why we are failing | 16:20 |
| fungi | betherly: looks like you just added reno support in a change that merged 14 hours ago | 16:20 |
| anteaya | pabelanger: thanks | 16:20 |
| anteaya | pabelanger: I'm interested in what you find | 16:20 |
| *** roxanaghe has joined #openstack-infra | 16:20 | |
| *** narayrak has quit IRC | 16:21 | |
| betherly | fungi: so if i submit another patch to ironic-ui it should build this time? | 16:22 |
| *** Cormite has quit IRC | 16:22 | |
| dhellmann | betherly, fungi : oh, I see a release notes job queued up now | 16:22 |
| *** Cormite has joined #openstack-infra | 16:22 | |
| dhellmann | betherly : it's not that the job didn't run, it's that it hasn't run yet | 16:22 |
| dhellmann | betherly : on http://status.openstack.org/zuul/ if you put "ironic-ui" in the filter you'll be able to see the job in the post queue | 16:23 |
| betherly | dhellmann: ah right ok! so once it has gone through there it should pass the release tests | 16:23 |
| dhellmann | right | 16:23 |
| *** Cormite has quit IRC | 16:23 | |
| dansmith | clarkb: this job failed because the tempest test that checks for multiple compute nodes only found one. Shouldn't there be a subnode directory in the logs if we really started the other node? http://logs.openstack.org/81/355981/3/check/gate-oslo.versionedobjects-src-grenade-dsvm-multinode/6115eb4/logs/ | 16:23 |
| *** Cormite has joined #openstack-infra | 16:24 | |
| pmalik | yolanda, mordred, pabelanger : Hello. Could somebody possibly take a look at new nv jobs for Trove project: https://review.openstack.org/#/c/354881/ We have been looking forward to improved test coverage for a while. Thank you. | 16:24 |
| dansmith | clarkb: specifically, I'm not sure where to look for failures setting up the other node | 16:24 |
| fungi | yeah, it looks like the merge commit that added reno support, dac3399, has made it out of the release pipeline already but probably predates the job getting added to the zuul layout? | 16:24 |
| clarkb | dansmith: I would have to read the logs myself. In theory not being able to talk to the second node should cause the job to fail fairly early | 16:24 |
| dhellmann | fungi : I think they were done in the right order, it's just that the post job to actually build the first set of release notes hasn't made it to the front of the queue yet | 16:24 |
| Shrews | pabelanger: yes, fix for unparseable is in. if you see it now, it's most likely a timeout (and a fix to ansible to clarify that it's really a timeout has been committed upstream) | 16:25 |
| Shrews | pabelanger: that one looks like a legit timeout | 16:25 |
| dansmith | clarkb: okay, but ... for a multinode job, there should be a subnode-2 directory in there right? | 16:25 |
| fungi | dhellmann: yeah, which seems to be for a later commit than the dac3399 merge commit | 16:25 |
| *** ihrachys has joined #openstack-infra | 16:25 | |
| clarkb | dansmith: not if it failed to copy the files | 16:25 |
| dhellmann | fungi : ah | 16:25 |
| dansmith | clarkb: well, yes, that's my point :) | 16:25 |
| clarkb | dansmith: and grenade does this funny basically doign the things devstack-gate should be doing... so I need to read through it to figure out what is going on | 16:25 |
| dansmith | clarkb: I'll just recheck, just trying to do my diligence | 16:26 |
| Shrews | pabelanger: TASK [zuul_runner with 1756 second timeout] -> 29 minutes, and failed at 30m | 16:26 |
| pabelanger | clarkb: so, we have a new project in tripleo-test-cloud-rh1 called openstackzuul, I've confirmed everything is setup properly with shade. We want this so nodepool.o.o doesn't see the private infrastructure that tripleo-ci is running. Basically, I'm ready to update the passwords in clouds.yaml for nodepool.o.o but wanted to check with you if I should do anything else? It's the same settings in | 16:26 |
| pabelanger | nodepool.yaml, just new credentials | 16:26 |
| dansmith | clarkb: 2016-08-16 19:20:01.066981 | [WARNING]: provided hosts list is empty, only localhost is available | 16:27 |
| pabelanger | Shrews: Ya, it does. Odd that ansible didn't report it correctly | 16:27 |
| Shrews | anteaya: also, see my comments ^^^ | 16:27 |
| * anteaya looks at Shrews comments | 16:27 | |
| dansmith | clarkb: and SUBNODES= at the beginning | 16:27 |
| Shrews | pabelanger: like i said, fix for that is in | 16:27 |
| clarkb | dansmith: neat thats probably the reason then. | 16:27 |
| Shrews | upstream only | 16:27 |
| dansmith | clarkb: okay | 16:27 |
| pabelanger | Shrews: okay cool. So we need to update ansible on zuul-launcher then | 16:27 |
| clarkb | pabelanger: you'll need to make sure that nodepool makes a new provider connection which it should do it cloud credentials change. Just double check that the new account is used afterwards | 16:27 |
| *** dprince has joined #openstack-infra | 16:27 | |
| fungi | dhellmann: betherly: confirmed, the dac3399 commit didn't run any release notes job because the change to add the job ( https://review.openstack.org/356029 ) didn't merge until ~8 hours ago | 16:27 |
| anteaya | pabelanger: Shrews I thought we did yesterday | 16:27 |
| mordred | pabelanger: I think jeblair has a playbook to do that | 16:27 |
| mtreinish | dansmith: shouldn't http://logs.openstack.org/81/355981/3/check/gate-oslo.versionedobjects-src-grenade-dsvm-multinode/6115eb4/console.html#_2016-08-16_18_26_10_100664 have 2-node in the node name if it's multinode? | 16:28 |
| mordred | anteaya: oh did we? | 16:28 |
| Shrews | pabelanger: well, the ansible with that is not released yet. you'd have to manually add the patch | 16:28 |
| anteaya | I had thought jeblair did so yesterday | 16:28 |
| clarkb | pabelanger: and you may need to do manual cleanup of db and old account since nodepool won't be able to delete things there at that point | 16:28 |
| betherly | fungi: dhellmann thanks for all your help!! | 16:28 |
| dansmith | mtreinish: no idea | 16:28 |
| pabelanger | mordred: Right, I think we want to graceful stop not to affect jobs | 16:28 |
| pabelanger | clarkb: yes, manually will be needed. | 16:28 |
| *** ihrachys has quit IRC | 16:28 | |
| mtreinish | dansmith: like: http://logs.openstack.org/11/348511/3/check/gate-grenade-dsvm-multinode/ad22cb4/console.html.gz#_2016-07-30_17_58_55_221814 | 16:28 |
| clarkb | mordred: see dansmith's thing, did shade change behavior around the IPs or similar? | 16:28 |
| Shrews | anteaya: 2 different ansible issues. we are running the first, not the second (with the proper log message) | 16:28 |
| pabelanger | Shrews: I think we are running stable-2.1 now | 16:28 |
| pabelanger | need to check | 16:29 |
| anteaya | monday: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-08-15.log.html#t2016-08-15T20:34:13 | 16:29 |
| jeblair | fungi, pabelanger: snafu with the entropy pump -- because the python folks "fixed" os.urandom in 3.5.2, it doesn't block. which means we can't detect from python whether getrandom is blocking. | 16:29 |
| anteaya | Shrews: ah sorry, my confusion | 16:29 |
| clarkb | mtreinish: oh good catch I think tahts actually the problem | 16:29 |
| dansmith | mtreinish: okay, this is on ovh, but I dunno if that's an important signal or not | 16:29 |
| Shrews | pabelanger: but which commit? | 16:29 |
| clarkb | mordred: ^ nevermind we ran it on the wrong label type | 16:29 |
| mordred | clarkb: cool | 16:29 |
| mordred | clarkb: there have been some ip address fixes recently - but I do not expect any of them to affect this | 16:29 |
| pabelanger | clarkb: okay, I'll schedule the change over with tripleo folks | 16:29 |
| Shrews | pabelanger: b/c jeblair set it up to get the first fix. i don't think we've pulled the 2nd | 16:29 |
| jeblair | Shrews, pabelanger: yes, that's correct | 16:30 |
| *** asettle has quit IRC | 16:30 | |
| pabelanger | Shrews: right, that's what I need to check | 16:30 |
| pabelanger | thanks | 16:30 |
| fungi | jeblair: can we by trying to read from /dev/urandom directly? | 16:30 |
| jeblair | fungi: turns out that still doesn't block, only 'getrandom' (which reads from that pool by default) blocks | 16:30 |
| pabelanger | Shrews: jeblair okay, so we could schedule an upgrade then. Let me find some time to do it | 16:30 |
| jeblair | fungi: so afaict, the only way to detect this state is using the getrandom syscall | 16:31 |
| fungi | jeblair: ohh... so we actually need to perform that syscall, which means a small c (or similar) wrapper? does cffi help for this? | 16:31 |
| jeblair | fungi: i was just about to look into that | 16:31 |
| *** Cormite has quit IRC | 16:32 | |
| *** Cormite has joined #openstack-infra | 16:32 | |
| clarkb | dansmith: mtreinish ya the job is set up wrong, patch in a sec | 16:32 |
| dansmith | clarkb: ah, so it's going to keep failing until we fix config? | 16:32 |
| jeblair | mordred: how are your cffi skillz? | 16:33 |
| clarkb | dansmith: yes | 16:34 |
| azvyagintsev | https://review.openstack.org/#/c/353861/ folks. what about merge ? (fungi - thx for yesterday help! ) | 16:34 |
| Shrews | pabelanger: the bigger issue is why didn't that rsync finish within 20m? Do we need to increase timeouts? | 16:34 |
| Shrews | s/20m/30m/ | 16:34 |
| dansmith | clarkb: okay, sorry I just rechecked then :/ | 16:34 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run ovo multinode grenade on 2 node label https://review.openstack.org/356592 | 16:34 |
| clarkb | mtreinish: dansmith ^ that should fix it | 16:34 |
| mordred | jeblair: I'd have to go read things | 16:34 |
| Shrews | pabelanger: oh, nm. it wasn't rsync. i was looking at the wrong line | 16:34 |
| jeblair | mordred, fungi: or maybe ctypes | 16:34 |
| dansmith | clarkb: thanks homey | 16:34 |
| jeblair | i'll see if i can do it with ctypes | 16:35 |
| mordred | jeblair: or cython | 16:35 |
| greghaynes | jeblair: if youre just doing a syscall ctypes is pretty easy | 16:35 |
| anteaya | clarkb: confirming you mean this fix for oslo.verstionedobjects? | 16:35 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Make task notes read-only by default https://review.openstack.org/325294 | 16:35 |
| clarkb | anteaya: yes | 16:35 |
| mordred | but yeah - ctypes is likely fine | 16:35 |
| greghaynes | jeblair: its what ive done in the past and 'just works' | 16:35 |
| anteaya | clarkb: thank you | 16:35 |
| *** ggnel_t has quit IRC | 16:35 | |
| jeblair | greghaynes: cool; i'll give it a shot and ping you on the review :) | 16:35 |
| *** jpich has quit IRC | 16:35 | |
| Shrews | jeblair: you know what'd be neat? if we could modify the ansible output on the fly to contain html links to the scripts it was running :) | 16:36 |
| *** ggnel_t has joined #openstack-infra | 16:36 | |
| jeblair | Shrews: d00d | 16:36 |
| mordred | Shrews: dood | 16:37 |
| fungi | wow... our meetbot's log isn't rotated | 16:38 |
| Shrews | lol | 16:38 |
| fungi | 2523053 loglines since 2012-04-27 | 16:38 |
| *** tphummel has joined #openstack-infra | 16:38 | |
| *** rwsu has quit IRC | 16:38 | |
| *** florianf has quit IRC | 16:39 | |
| *** shamail has joined #openstack-infra | 16:39 | |
| anteaya | ha ha ha | 16:39 |
| *** shashank_hegde has joined #openstack-infra | 16:39 | |
| *** gyee has joined #openstack-infra | 16:39 | |
| *** ihrachys has joined #openstack-infra | 16:41 | |
| *** spzala has quit IRC | 16:41 | |
| clarkb | fungi: cloudnull for that sysctl ipv6 stuff, was use_tempaddr set to 1 or 2? | 16:41 |
| cloudnull | ubuntu defaults to 2 | 16:42 |
| *** spzala has joined #openstack-infra | 16:42 | |
| clarkb | wow ok | 16:42 |
| fungi | clarkb: good question, i didn't check, i only tested setting it to 0 which did seem to resolve the traceroue6 issue | 16:42 |
| pabelanger | clarkb: regarding the credential changes, I would expect the running jobs to continue until finished right? | 16:42 |
| greghaynes | clarkb: http://paste.ubuntu.com/23065130/ looks like what gets set by default in ubuntu cloudimages | 16:43 |
| clarkb | pabelanger: ya then nodepool won't be able to delete their hosts after the jobs finish because it won't have access to those instances. This is the manual cleanup part | 16:43 |
| greghaynes | clarkb: so I guess dib should do the same | 16:43 |
| fungi | clarkb: cloudnull: it _may_ be that 1 is sufficient, and perhaps trusty defaulted to 2 but xenial switched that to 1 (which would explain why we didn't have problems on xenial) | 16:43 |
| pabelanger | clarkb: right, thanks. What I expect too | 16:43 |
| *** pilgrimstack has quit IRC | 16:43 | |
| clarkb | fungi: cloudnull yes I think 1 is sufficient but 0 is the kernel default so that value seems like a good default :) | 16:43 |
| fungi | clarkb: i've always felt like the v6 privacy extensions were a security by obscurity crutch anyway. never been a fan | 16:44 |
| cloudnull | yea, setting it to 1 would likely make it go | 16:44 |
| *** ihrachys has quit IRC | 16:44 | |
| clarkb | but if kernel and ubuntu cloud images all agree 0 is a good default I say we start there (whcih is what we did right?) | 16:45 |
| cloudnull | the xenial cloud image ships with /etc/sysctl.d/99-cloudimg-ipv6.conf as well though | 16:45 |
| clarkb | just making sure I understand what was going on | 16:45 |
| openstackgerrit | Joe Talerico proposed openstack-infra/system-config: Browbeat upstream mailer https://review.openstack.org/356601 | 16:45 |
| *** martinkopec has quit IRC | 16:45 | |
| fungi | sure, it helps stop leaking "some" information like what network hardware you have and makes it much harder to enumerate address space, but it also makes it a lot more complicated to troubleshoot your network | 16:46 |
| *** edtubill has quit IRC | 16:46 | |
| *** raunak has joined #openstack-infra | 16:46 | |
| *** ihrachys has joined #openstack-infra | 16:46 | |
| *** spzala has quit IRC | 16:46 | |
| cloudnull | setting to 1 would allow it to use either path. but to get thing going i just went with the hammer route. | 16:46 |
| fungi | our test nodes don't need more than one global v6 address anyway, so wfm | 16:47 |
| cloudnull | quick test on one of my baremetal nodes shows xenial uses "net.ipv6.conf.all.use_tempaddr = 2" and "net.ipv6.conf.default.use_tempaddr = 2" by default too. | 16:49 |
| *** dtantsur|brb is now known as dtantsur | 16:49 | |
| *** ihrachys has quit IRC | 16:49 | |
| cloudnull | so getting adding that conf file should make our cloud images happier w/ ipv6 | 16:49 |
| *** ihrachys has joined #openstack-infra | 16:49 | |
| *** ldnunes has quit IRC | 16:49 | |
| *** _nadya_ has joined #openstack-infra | 16:50 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Corrected success-pattern for npm-based doc builds https://review.openstack.org/356606 | 16:51 |
| krotscheck | Quick typo fix, if anyone's so inclined to look at it... ^^ | 16:51 |
| *** Jeffrey4l has quit IRC | 16:52 | |
| *** _nadya_ has quit IRC | 16:53 | |
| openstackgerrit | Zsolt Dudás proposed openstack-infra/jenkins-job-builder: Replaces logging with oslo.log https://review.openstack.org/354850 | 16:53 |
| cloudnull | clarkb fungi: i PR'd the changes into DIB https://review.openstack.org/#/c/356456 too | 16:53 |
| anteaya | <-- offline for a bit | 16:53 |
| *** ldnunes has joined #openstack-infra | 16:53 | |
| pabelanger | #status log tripleo-test-cloud-rh1 credentials update on nodepool.o.o to use opentackzuul project | 16:54 |
| *** pabelanger has quit IRC | 16:54 | |
| *** pabelanger has joined #openstack-infra | 16:54 | |
| clarkb | cloudnull: looks like centos may not have liked that change | 16:54 |
| pabelanger | #status log tripleo-test-cloud-rh1 credentials update on nodepool.o.o to use opentackzuul project | 16:54 |
| openstackstatus | pabelanger: finished logging | 16:54 |
| clarkb | cloudnull: wouldn't be surprised if that /etc/sysctl.d doesn't exist on centos or something like that | 16:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added npm-docs draft build for eslint-config-openstack. https://review.openstack.org/356609 | 16:55 |
| krotscheck | betherly: ^^ | 16:55 |
| *** rbuzatu has quit IRC | 16:55 | |
| cloudnull | clarkb: it looks like a repo failure | 16:55 |
| cloudnull | http://logs.openstack.org/56/356456/1/check/gate-dib-dsvm-functests-centos-7/2f4b6db/console.html#_2016-08-17_13_55_07_296838 | 16:55 |
| cloudnull | on a deb ... | 16:55 |
| clarkb | cloudnull: gah, I guess recheck is the next step then | 16:56 |
| *** sambetts is now known as sambetts|afk | 16:56 | |
| clarkb | greghaynes: ^ you may be interested in that too | 16:56 |
| cloudnull | i guess that still constructing the build env | 16:56 |
| *** sandanar has joined #openstack-infra | 16:56 | |
| cloudnull | yup, just rechecked. | 16:56 |
| clarkb | cloudnull: ya dib does the full cross join test to amek sure centos can build debuntu and debuntu can build centos | 16:56 |
| greghaynes | yea, mirror fail :( | 16:56 |
| pabelanger | surprised we aren't using our mirrors for those jobs | 16:56 |
| clarkb | pabelanger: I think it just need work to add them to the tests | 16:57 |
| greghaynes | that needs a debian mirror | 16:57 |
| clarkb | pabelanger: since its the full cross join situation | 16:57 |
| greghaynes | do we have one of those? | 16:57 |
| pabelanger | greghaynes: yes | 16:57 |
| greghaynes | oh awesome | 16:57 |
| greghaynes | yes, that just needs work then | 16:57 |
| pabelanger | nice | 16:57 |
| *** nwkarsten has quit IRC | 16:57 | |
| *** yamahata has quit IRC | 16:58 | |
| *** nwkarsten has joined #openstack-infra | 16:58 | |
| *** weshay_bbiab is now known as weshay | 16:58 | |
| *** vgridnev has joined #openstack-infra | 16:59 | |
| *** liusheng has quit IRC | 16:59 | |
| *** lucasagomes is now known as lucas-dinner | 17:00 | |
| *** tesseract- has quit IRC | 17:00 | |
| *** notnownikki has quit IRC | 17:01 | |
| *** sandanar_ has joined #openstack-infra | 17:01 | |
| *** nwkarsten has quit IRC | 17:02 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 17:03 |
| *** shashank_hegde has quit IRC | 17:04 | |
| *** sandanar has quit IRC | 17:04 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Nodejs6 jobs are now voting. https://review.openstack.org/356616 | 17:04 |
| *** rbuzatu has joined #openstack-infra | 17:05 | |
| *** e0ne has quit IRC | 17:05 | |
| openstackgerrit | Adam Reznechek proposed openstack-infra/project-config: Add networking-powervm doc publishing to RTFD https://review.openstack.org/356619 | 17:05 |
| *** tphummel has quit IRC | 17:06 | |
| *** Hal has quit IRC | 17:07 | |
| *** Hal has joined #openstack-infra | 17:08 | |
| *** jerryz has joined #openstack-infra | 17:08 | |
| *** acoles is now known as acoles_ | 17:08 | |
| *** jkilpatr_ has joined #openstack-infra | 17:08 | |
| *** jkilpatr has quit IRC | 17:09 | |
| *** tphummel has joined #openstack-infra | 17:10 | |
| *** Hal1 has joined #openstack-infra | 17:11 | |
| pabelanger | clarkb: what did we decided about the ipv6 firewall issue for multinode jobs in osic? | 17:13 |
| *** inc0 has quit IRC | 17:13 | |
| *** Hal has quit IRC | 17:15 | |
| *** yaume has joined #openstack-infra | 17:15 | |
| clarkb | we need to use iptables6 | 17:17 |
| *** Sukhdev has quit IRC | 17:17 | |
| pabelanger | okay, let me get a patch up for that | 17:17 |
| azvyagintsev | folks, could you please review/merge https://review.openstack.org/#/c/353861/ ? | 17:18 |
| *** rbuzatu has quit IRC | 17:18 | |
| *** ldnunes has quit IRC | 17:20 | |
| *** roxanaghe has quit IRC | 17:21 | |
| *** tqtran has joined #openstack-infra | 17:22 | |
| *** asettle has joined #openstack-infra | 17:23 | |
| *** kzaitsev_mb has quit IRC | 17:23 | |
| *** rbuzatu has joined #openstack-infra | 17:24 | |
| *** ldnunes has joined #openstack-infra | 17:24 | |
| *** amotoki has quit IRC | 17:24 | |
| *** ihrachys has quit IRC | 17:25 | |
| *** amotoki has joined #openstack-infra | 17:25 | |
| pabelanger | clarkb: is ip6tables smart enough to deal with both ipv4/ipv6 address? or do we need to update nodepool to store those addresses in different files | 17:25 |
| pabelanger | testing on a node now | 17:25 |
| *** tonytan4ever has quit IRC | 17:25 | |
| *** rcernin has joined #openstack-infra | 17:26 | |
| clarkb | you need to switch on address type iiec | 17:26 |
| clarkb | you could probavly just do if : in ipaddrs then use ip6tables else use iptables | 17:26 |
| pabelanger | clarkb: thanks, testing | 17:26 |
| *** tqtran has quit IRC | 17:27 | |
| *** electrofelix has quit IRC | 17:27 | |
| *** amotoki has quit IRC | 17:28 | |
| *** asettle has quit IRC | 17:28 | |
| dansmith | mtreinish: clarkb: just to be clear, I can't depends-on that project-config change right? | 17:29 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/infra-specs: Add survey spec https://review.openstack.org/349831 | 17:29 |
| clarkb | dansmith: you can but ut wont test with it, the dependson in that case will just order the merges | 17:29 |
| dansmith | clarkb: yeah, okay | 17:29 |
| *** armax has quit IRC | 17:31 | |
| *** Rockyg has quit IRC | 17:32 | |
| *** yamamoto has joined #openstack-infra | 17:34 | |
| *** klindgren has quit IRC | 17:35 | |
| *** tqtran has joined #openstack-infra | 17:35 | |
| rcarrillocruz | greghaynes: heya, i tried to rebuild the deploy image on bifrost and getting this error http://paste.openstack.org/show/559075/ | 17:35 |
| *** priteau has quit IRC | 17:36 | |
| rcarrillocruz | if I read that right, there should be a dupe *somewhere* on the ironic-agent element service | 17:36 |
| rcarrillocruz | but i don't see it | 17:36 |
| *** dtantsur is now known as dtantsur|afk | 17:37 | |
| *** yamamoto has quit IRC | 17:38 | |
| *** shamail has quit IRC | 17:38 | |
| *** yamahata has joined #openstack-infra | 17:41 | |
| pabelanger | clarkb: https://review.openstack.org/356637 | 17:42 |
| fungi | did somebody just restart nodepool or something? i saw a bunch of running xenial jobs in the gate suddenly requeue and start over, with no gate reset or changes failing | 17:42 |
| openstackgerrit | Merged openstack-dev/hacking: Add optional H203 to check that assertIs(Not)None is used https://review.openstack.org/276517 | 17:42 |
| greghaynes | rcarrillocruz: lets chat in #openstack-dib | 17:42 |
| rcarrillocruz | oh nice, there's a channel :-) | 17:43 |
| greghaynes | yep :) | 17:43 |
| *** stevemar_ has joined #openstack-infra | 17:43 | |
| *** stevemar_ has quit IRC | 17:43 | |
| *** tphummel has quit IRC | 17:45 | |
| *** markusry has joined #openstack-infra | 17:47 | |
| *** hockeynut has quit IRC | 17:48 | |
| jlvillal | sdague: Not sure what you think about this PS4 change for devstack-gate https://review.openstack.org/#/c/271434/ | 17:48 |
| jlvillal | sdague: I was unsure if you would prefer to have the short_source function from devstack in there. Or if simple is okay. | 17:48 |
| Shrews | devstack-gate run on a PS4? :) | 17:49 |
| jlvillal | Shrews: I want it to :) | 17:49 |
| jlvillal | Shrews: No. PS4 is an environment variable for bash | 17:49 |
| Shrews | i'm so baller at the devstack-gate game. level 17 | 17:49 |
| Shrews | jlvillal: i know. just being silly | 17:49 |
| *** shashank_hegde has joined #openstack-infra | 17:50 | |
| dansmith | Shrews: I got it and chuckled :) | 17:50 |
| jlvillal | Shrews: You can set PS4 to show things like what the file name is, line number, function name. I find it very useful :) | 17:50 |
| jlvillal | Shrews: heh | 17:50 |
| *** tosky has quit IRC | 17:51 | |
| *** roxanaghe has joined #openstack-infra | 17:51 | |
| *** inc0 has joined #openstack-infra | 17:52 | |
| *** spzala has joined #openstack-infra | 17:53 | |
| nibalizer | need to pay for playstation online in order to run openstack | 17:54 |
| *** pt_15 has quit IRC | 17:55 | |
| clarkb | pabelanger: I need to test if that grep will work since : means function or something in bash (so might need to be escaped) but otherwise that looks about right | 17:55 |
| pabelanger | clarkb: Ya, tested locally it worked | 17:56 |
| pabelanger | but we can escape to be safe | 17:56 |
| *** nwkarsten has joined #openstack-infra | 17:56 | |
| *** jcoufal has quit IRC | 17:56 | |
| *** nwkarsten has quit IRC | 17:56 | |
| *** pt_15 has joined #openstack-infra | 17:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove python3 jobs for zuul https://review.openstack.org/355239 | 17:57 |
| clarkb | pabelanger: seems to work here too, I am guessing the parser is smart about where : is valid as a non string token | 17:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove python3 jobs from nodepool https://review.openstack.org/355449 | 17:57 |
| *** nwkarsten has joined #openstack-infra | 17:57 | |
| mordred | nibalizer: my TV has playstation built in - does that mean I could run OpenStack on my TV? | 17:57 |
| *** markusry has quit IRC | 17:58 | |
| *** spzala has quit IRC | 17:58 | |
| *** jcoufal has joined #openstack-infra | 17:59 | |
| fungi | r.i.p. wulfstation project | 17:59 |
| nibalizer | mordred: subscrbtion based model | 17:59 |
| *** hockeynut has joined #openstack-infra | 17:59 | |
| Zara | depends if it's a PS1 | 18:00 |
| *** spzala has joined #openstack-infra | 18:00 | |
| Shrews | openstack on tvs? think of the possibilities? i wanna propose a new project called Incredibly Distributed Internet of Things | 18:00 |
| fungi | nibalizer: your ideas are intriguing to me and i wish to subscribe to your openstack | 18:00 |
| Shrews | openstack/idiot for short | 18:01 |
| fungi | Shrews: best acronym evar | 18:01 |
| pleia2 | haha | 18:01 |
| mordred | I think Shrews just won openstack | 18:01 |
| mordred | we can all go home now | 18:01 |
| openstackgerrit | amrith proposed openstack-infra/project-config: Update trove jobs to include api-ref job https://review.openstack.org/356407 | 18:01 |
| *** tphummel has joined #openstack-infra | 18:02 | |
| nibalizer | wow | 18:02 |
| fungi | also would consider "extremely efficient distributed internet of things" | 18:02 |
| *** markusry has joined #openstack-infra | 18:03 | |
| dansmith | clarkb: who else can I bribe to merge that config change? | 18:03 |
| *** javeriak has joined #openstack-infra | 18:03 | |
| fungi | (acronym spoken in ren höek's voice) | 18:03 |
| dansmith | fungi: https://review.openstack.org/#/c/356592/ <-- blocking a fix in an oslo library | 18:03 |
| Shrews | fungi: i never read it any other way | 18:03 |
| mordred | dansmith: wich one? | 18:03 |
| mordred | ah | 18:03 |
| clarkb | dansmith: any of the project-config core? anteaya fungi mordred pleia2 nibalizer jeblair pabelanger rcarrillocruz have all been active recently | 18:03 |
| dansmith | clarkb: are those ordered in most likely to take a bribe order? | 18:04 |
| *** jordanP has joined #openstack-infra | 18:04 | |
| clarkb | dansmith: no order is clarkb typing as he remembered :) | 18:04 |
| fungi | thanks, was trying to find the change numbe rin scrollback | 18:04 |
| pleia2 | depends on what you're offering | 18:04 |
| dansmith | clarkb: :) | 18:04 |
| fungi | oh, mordred has already smacked it with his +3 hammer of approval | 18:05 |
| dansmith | pleia2: well, nothing because mordred did it without bargaining.. | 18:05 |
| dansmith | heh | 18:05 |
| mordred | dansmith: I expect that some point, I'll need a favor, and I'll know who to ask | 18:05 |
| rcarrillocruz | clarkb: mordred was faster | 18:05 |
| dansmith | oh crap, the favor-owed future | 18:05 |
| dansmith | those are dangerous | 18:05 |
| * mordred has been practicing to set up his own mafia | 18:05 | |
| dansmith | next thing I know I'll be swearing at mordred's TV-with-a-PS4 trying to get devstack running | 18:05 |
| clarkb | pabelanger: another thing to test related to that ready script change is whether or not devstack-gate has any ipv4 assumptions (and I think it does at least assume the mtu | 18:06 |
| pleia2 | haha | 18:06 |
| * fungi tries to figure out how many favors we owe dansmith for keeping nova from falling apart at the seams | 18:06 | |
| dansmith | heh | 18:06 |
| jeblair | we are, i would say, on average, pool at bargaining. | 18:07 |
| jeblair | poor even | 18:07 |
| mordred | jeblair: I think we're pool | 18:07 |
| jeblair | i've been typing pool a lot lately | 18:07 |
| *** pvaneck has joined #openstack-infra | 18:07 | |
| clarkb | pabelanger: let me check some of that stuff in d-g | 18:07 |
| * jeblair takes two drinks | 18:07 | |
| mordred | jeblair: :) | 18:07 |
| *** _nadya_ has joined #openstack-infra | 18:07 | |
| *** alexz__ has quit IRC | 18:07 | |
| *** javeriak_ has joined #openstack-infra | 18:07 | |
| *** javeriak has quit IRC | 18:07 | |
| *** dkehn has quit IRC | 18:07 | |
| *** dkehn_ has quit IRC | 18:08 | |
| *** tqtran has quit IRC | 18:09 | |
| pabelanger | clarkb: ack | 18:09 |
| *** _nadya_ has quit IRC | 18:12 | |
| *** xarses_ has quit IRC | 18:13 | |
| *** mtanino has quit IRC | 18:13 | |
| *** sandanar_ has quit IRC | 18:13 | |
| clarkb | pabelanger: looks like vxlan wasn't supported in linux until 3.11? and ovs doesn't support it at all? | 18:13 |
| * clarkb grumbles | 18:14 | |
| clarkb | doesn't look like ovs can gre ipv6 either | 18:14 |
| openstackgerrit | Merged openstack-infra/storyboard: Fix filtering by Board title https://review.openstack.org/356542 | 18:14 |
| *** dkehn has joined #openstack-infra | 18:15 | |
| clarkb | mordred: ^ so, are we not putting the ipv4 private addr in the private address list with osic? | 18:16 |
| clarkb | mordred: because it turns out we probably actually need that to happen to have multinode work properly | 18:16 |
| mordred | blast | 18:16 |
| mordred | we are not - and I deferred fixing it because of thinking that it did not matter | 18:17 |
| mordred | but if it does matter, we can fix it | 18:17 |
| clarkb | I think it does matter because for some hard to understand reason tools like ovs don't support tunneling over ipv6 | 18:17 |
| mordred | warning: the review of the patch(s) to fix it are going to likely hurt someone's head | 18:17 |
| clarkb | they probably assume you won't need to tunnel because ipv6? | 18:17 |
| mordred | clarkb: that's what I'd assume :) | 18:17 |
| mordred | clarkb: I'll try to sketch up an approach soon | 18:18 |
| * mordred goes off to cry in advance for a little bit first | 18:18 | |
| clarkb | as much as I wish ipv6 meant you won't need tunnels, that seems like a bad assumption when clouding | 18:19 |
| *** tqtran has joined #openstack-infra | 18:19 | |
| *** dkehn_ has joined #openstack-infra | 18:20 | |
| *** ldnunes has quit IRC | 18:22 | |
| cloudnull | clarkb: 3.10 from RHEL supports vxlan, otherwise i think you have to go to 3.13+ | 18:23 |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo with AFS_DIR https://review.openstack.org/356546 | 18:23 |
| clarkb | cloudnull: ya but thats only with linux bridge not ovs | 18:23 |
| clarkb | and we use ovs for raisins | 18:23 |
| cloudnull | ha, fair, likely missed that context :'( | 18:24 |
| *** ociuhandu has quit IRC | 18:24 | |
| clarkb | (linux bridge can only do point to point vxlan without multicast) | 18:24 |
| *** tonytan4ever has joined #openstack-infra | 18:26 | |
| *** narayrak has joined #openstack-infra | 18:29 | |
| *** yaume has quit IRC | 18:30 | |
| *** ldnunes has joined #openstack-infra | 18:31 | |
| clarkb | adam_g: have you run into ^ with your networking stuff? any insights or suggestions? | 18:31 |
| clarkb | adam_g: tl;dr is can we run tunnels over ipv6 with GRE? | 18:31 |
| *** tonytan4ever has quit IRC | 18:31 | |
| clarkb | adam_g: using ovs specifically | 18:34 |
| clarkb | (or with vxlan) | 18:34 |
| amrith | hi folks, when a CI machine is only accessible using an IPv6 address (on zuul's status page), I can't seem to reach that host. Is there something that I need to change or is it an infra issue? | 18:35 |
| amrith | take change 356631 for example. | 18:35 |
| mordred | amrith: do you have IPv6 connectivity locally? | 18:35 |
| amrith | I can't get to telnet://2001:4800:1ae1:18:f816:3eff:fe33:fd38:19885 | 18:36 |
| amrith | mordred, yes | 18:36 |
| *** yaume has joined #openstack-infra | 18:36 | |
| mordred | hrm. is something parsing the port as part of the ipv6 address? | 18:36 |
| fungi | amrith: i have a feeling that url pattern may need tweaking in zuul to look like telnet://[2001:4800:1ae1:18:f816:3eff:fe33:fd38]:19885 but i haven't tested that yet | 18:36 |
| amrith | well, I just take that command line and drop it into a shell | 18:36 |
| jeblair | heh, yeah, probably so | 18:36 |
| *** Cormite has quit IRC | 18:36 | |
| amrith | telnet <ip> <port> | 18:36 |
| amrith | but that ma be the issue | 18:36 |
| *** yaume has quit IRC | 18:37 | |
| fungi | amrith: oh, your telnet client may not support ipv6 either? | 18:37 |
| clarkb | you can try nc | 18:37 |
| clarkb | but also double check ipv6 is working properly | 18:37 |
| fungi | amrith: i've been using the netcat-openbsd package on debian/ubuntu which handles ipv6 addresses just fine | 18:37 |
| fungi | nc 2001:4800:1ae1:18:f816:3eff:fe33:fd38 19885 | 18:37 |
| *** inc0 has quit IRC | 18:37 | |
| *** degorenko is now known as _degorenko|afk | 18:37 | |
| fungi | it's worth noting that this isn't _really_ a telnet server, it's just a tcp socket streaming text | 18:38 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 18:38 |
| fungi | telnet clients happen to deal with that pretty well, but there's no actual telopt negotiation or anything like that | 18:38 |
| amrith | fungi, I'll try nc | 18:38 |
| fungi | nor does it probably correctly implement telnet escaping | 18:38 |
| amrith | will check ... thx mordred fungi clarkb | 18:38 |
| fungi | (the escape byte is special in telnet protocol as it's used to signal control commands inline, so has to be doubled or telnet clients tend to freak out) | 18:39 |
| *** tqtran has quit IRC | 18:40 | |
| fungi | telnet:// just happened to be a convenient shorthand approximation for what these console streams are | 18:40 |
| amrith | fungi, I noticed that my client responded properly to ^] when I wanted to close; that's the only thing I cared about. It tailed the log nicely (when the IP was v4). but will try again on the next job I run. | 18:41 |
| fungi | ctrl-c will abort a netcat session | 18:41 |
| *** tqtran has joined #openstack-infra | 18:43 | |
| mtreinish | fungi: heh, well it assumes you have an ipv6 route configured on your machine | 18:44 |
| mtreinish | which apparently I don't on my laptop... | 18:44 |
| fungi | no v6 router advertisements from your gateway? | 18:45 |
| mtreinish | I'm at starbucks, but not that I can see | 18:45 |
| fungi | oh | 18:45 |
| *** nwkarsten has quit IRC | 18:45 | |
| *** rbuzatu has quit IRC | 18:45 | |
| fungi | well, *$ likely considers ipv6 old and stodgy and has moved on to the hip new ipv7 | 18:46 |
| fungi | it comes with a goatee and skinny jeans | 18:46 |
| mtreinish | heh, it was time for my daily escape from the inadequately cooled apartment | 18:46 |
| *** nwkarsten has joined #openstack-infra | 18:46 | |
| mtreinish | haha, I could see that | 18:46 |
| fungi | at home we're managing to maintain a -20f differential from outside, which is starting to get on the uncomfortable side when the ambient outside temperatures here reach 100f | 18:47 |
| fungi | after several weeks straight of this, i'm ready for fall | 18:48 |
| mtreinish | fungi: well I've started playing with trying to make it better: http://blog.kortar.org/?p=319 but the problem is my desk at home is as far away from the ACs as it can be (well without being in the kitchen or bathroom) | 18:49 |
| mtreinish | and the in wall units don't do a good job of cooling down that far away | 18:49 |
| *** woodster_ has quit IRC | 18:49 | |
| *** nwkarsten has quit IRC | 18:50 | |
| pabelanger | mordred: did we want to try restarting nodepool again to pick up the latest shade fixes? | 18:51 |
| fungi | mtreinish: neat. i take it you also ruled out servo controls to drive the knobs? | 18:51 |
| fungi | mechanically instead of electronically, so you could have just molded a driver to fit the outside indentations of the knobs | 18:52 |
| *** nwkarsten has joined #openstack-infra | 18:52 | |
| fungi | though that's starting to veer from home automation to robotics i guess | 18:52 |
| mordred | pabelanger: yah - the only fix that's there that we might not be running with right now is an optimization - but it's a good one to get in | 18:52 |
| mtreinish | fungi: hah, I thought about it (and several other people suggested it too) but it seemed like more effort than it was worth | 18:52 |
| fungi | fair enough | 18:52 |
| fungi | but... robots! | 18:52 |
| mtreinish | fungi: heh, yeah I was just typing that | 18:53 |
| mtreinish | I'd have home automation robots :) | 18:53 |
| fungi | my only home automation robot is a roomba, so it's not like i have room to talk | 18:53 |
| dansmith | mtreinish: I ripped the control board out of a 13k standing unit and replaced it with one of my custom HVAC controllers for the server room | 18:53 |
| dansmith | mtreinish: works nice | 18:53 |
| pabelanger | mordred: which one was that? | 18:54 |
| dansmith | mtreinish: it's nice to have an ethernet cable coming out of the back of your AC | 18:54 |
| mordred | pabelanger: "Deal with clouds that don't have fips betterer" | 18:54 |
| mordred | pabelanger: oh - sorry, that's the optimization | 18:54 |
| clarkb | with the recent heat in portalnd I just go to thermostat and set it to "On" and it runs the AC continuously | 18:54 |
| mordred | pabelanger: hopefully the next release will deal with the private/public ipv4 address in osic better | 18:55 |
| mordred | but I still have a patch to write | 18:55 |
| mtreinish | dansmith: cool, yeah that was the direction I wanted to go at first. But I don't actually own the ac units in my apt | 18:55 |
| pabelanger | mordred: Ah, okay. | 18:55 |
| *** devkulkarni has joined #openstack-infra | 18:55 | |
| clarkb | basically have fungi's problem. My temp diff at best is 18F which isn't enough for 100 degree days | 18:55 |
| dansmith | mtreinish: yeah, that's a problem :) | 18:55 |
| pabelanger | mordred: So, I have the bandwidth if we want to restart, your call | 18:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Make Launchpad credentials reusable https://review.openstack.org/356665 | 18:57 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add LP creds and launchpadlib to signing job node https://review.openstack.org/356666 | 18:57 |
| *** rkukura has quit IRC | 18:57 | |
| mordred | pabelanger: nah - I don't think it's urgent today | 18:58 |
| *** jamielennox has quit IRC | 19:06 | |
| *** edtubill has joined #openstack-infra | 19:06 | |
| *** fifieldt has quit IRC | 19:07 | |
| *** Sukhdev has joined #openstack-infra | 19:07 | |
| *** wznoinsk has quit IRC | 19:07 | |
| *** beagles is now known as beagles_biab | 19:08 | |
| *** matt-borland has quit IRC | 19:08 | |
| *** rkukura has joined #openstack-infra | 19:09 | |
| *** jamielennox has joined #openstack-infra | 19:09 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:10 | |
| *** gyee has quit IRC | 19:11 | |
| cloudnull | pabelanger mordred fungi: do we know if we can re-enable the trusty builds on OSIC or are we still waiting for the new images to come online? | 19:12 |
| *** wewe0901 has quit IRC | 19:13 | |
| *** tongli has quit IRC | 19:15 | |
| *** matt-borland has joined #openstack-infra | 19:15 | |
| pabelanger | ya, we need a new image | 19:16 |
| pabelanger | let me check | 19:16 |
| cloudnull | no rush. i was just curious | 19:16 |
| pabelanger | build failed it looks, I'll kick if off again | 19:17 |
| *** spzala has quit IRC | 19:18 | |
| *** fifieldt has joined #openstack-infra | 19:18 | |
| *** inc0 has joined #openstack-infra | 19:19 | |
| *** spzala has joined #openstack-infra | 19:19 | |
| fungi | pabelanger: probably choked due to the massive addition of packaging repos today | 19:19 |
| pabelanger | fungi: yup, looks like it | 19:19 |
| fungi | if a project makes it into the projects.yaml in project-config before ansible/puppet run create-cgitrepos on all our git server farm, then dib's source-repos fails to retrieve one of those into the image cache, and *boom* | 19:20 |
| *** _nadya_ has joined #openstack-infra | 19:21 | |
| *** _nadya_ has quit IRC | 19:21 | |
| *** spzala_ has joined #openstack-infra | 19:22 | |
| *** jkilpatr_ is now known as jkilpatr | 19:22 | |
| odyssey4me | oh dear, a bunch of POST_FAILURES - I take it that we have a volume corruption somewhere? | 19:22 |
| fungi | doesn't look like it | 19:23 |
| fungi | dmesg on static.o.o is pretty quiet | 19:23 |
| *** spzala has quit IRC | 19:23 | |
| odyssey4me | hmm, odd - http://logs.openstack.org/54/356554/1/check/gate-openstack-ansible-os_tempest-docs-ubuntu-xenial/9e60f45/ | 19:24 |
| odyssey4me | https://review.openstack.org/356554 - one sample | 19:24 |
| fungi | yeah, i'm going through them, looks like they were all xenial nodes afaict and none of them managed to get any logs uploaded | 19:24 |
| *** ociuhandu has joined #openstack-infra | 19:25 | |
| odyssey4me | that one has a centos failure too | 19:25 |
| odyssey4me | http://logs.openstack.org/54/356554/1/check/gate-openstack-ansible-os_tempest-ansible-func-centos-7-nv/515b33b/ | 19:25 |
| *** wznoinsk has joined #openstack-infra | 19:26 | |
| *** spzala_ has quit IRC | 19:26 | |
| fungi | oh! | 19:26 |
| fungi | /dev/mapper/main-logs 12T 12T 0 100% /srv/static/logs | 19:26 |
| pabelanger | wow | 19:27 |
| odyssey4me | whoops | 19:27 |
| fungi | i guess sdague wasn't joking when he said keystone job logging went off the rails | 19:27 |
| odyssey4me | lol, sorry - didn't mean to ping you whoops | 19:27 |
| *** ayoung has quit IRC | 19:27 | |
| sdague | fungi: nope, really wasn't :) | 19:27 |
| *** annegentle has quit IRC | 19:27 | |
| sdague | fungi: honestly, you might just have to delete keystone logs from the past 2 days | 19:28 |
| fungi | well, this was an upturn over the past 2 months... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 19:28 |
| *** edmondsw has quit IRC | 19:28 | |
| fungi | we spiked up from about 50% utilized in june | 19:28 |
| stevemar | eek :\ | 19:29 |
| mtreinish | fungi: oh, that reminds me we never did finish the cacti firehose cleanup did we? | 19:29 |
| sdague | fungi: ok, sure, but the keystone giant logs went on 2 days ago | 19:29 |
| *** xarses_ has joined #openstack-infra | 19:29 | |
| stevemar | we're reworking it with trace level | 19:29 |
| fungi | so might not be all keystone's doing | 19:29 |
| stevemar | sorry bout that :\ | 19:29 |
| EmilienM | in zuul, is all-files-match-any addressed for experimental jobs? | 19:29 |
| sdague | fungi: sure | 19:29 |
| EmilienM | or only for check/gate? | 19:29 |
| fungi | mtreinish: no, https://review.openstack.org/356090 should fix that once approved | 19:29 |
| sdague | anyway, the sooner we can purge something here, we could recover some of these patches | 19:30 |
| fungi | sdague: yep, i'm looking at early expiring some logs for now to get us some breathing room | 19:30 |
| sdague | coolio | 19:30 |
| mtreinish | fungi: there was also removing the firehose.o.o entry right? (since we changed it to firehose01.o.o) | 19:30 |
| fungi | mtreinish: yeah, but that has to be done manually i think | 19:31 |
| mtreinish | oh, ok | 19:31 |
| sdague | mtreinish: let fungi save the gate first :) | 19:31 |
| odyssey4me | fungi not sure if this is intentional, but I've noticed that not all logs get compressed when they get old | 19:32 |
| odyssey4me | an example - see http://logs.openstack.org/19/349619/1/gate/gate-openstack-ansible-dsvm-commit/9147857/ in the logs folder | 19:32 |
| pabelanger | zigo: can you try another package build | 19:33 |
| *** tphummel_ has joined #openstack-infra | 19:33 | |
| odyssey4me | it looks like in that case, only .txt and .html files are compressed | 19:33 |
| fungi | i'm dropping our log retention manually from 4 months to 3 for the moment, and then we can selectively clean up whatever else we think might buy us space | 19:33 |
| *** tphummel has quit IRC | 19:34 | |
| *** tphummel_ is now known as tphummel | 19:34 | |
| fungi | logs for the past 2 days only account for about 10% of the data there, so we want more breathing room than removing keystone logs is going to buy us regardless | 19:34 |
| *** xarses_ has quit IRC | 19:34 | |
| *** eandersson_ has quit IRC | 19:36 | |
| sdague | it's still impressive that it's 10% of the data | 19:37 |
| fungi | indeed | 19:37 |
| fungi | we're already up to almost 1gb freed back up so i think we're in the clear for any jobs ending now onward, but maybe i can save the current state of the integrated gate with a promote of the top change on it | 19:38 |
| sdague | ok, I'm going to sacrifice auggy's patch in the gate to reset and save the rest of what's behind it | 19:39 |
| *** kushal has quit IRC | 19:39 | |
| fungi | oh, i was just going to promote 356211,1 (which is already at the top) | 19:39 |
| fungi | which should requeue everything in there | 19:40 |
| sdague | well, I found the nova patch before the first fail and rebased it | 19:40 |
| fungi | but looks like you beat me to it | 19:40 |
| auggy | my patch will gladly be sacrificed to appease the gate gods | 19:40 |
| fungi | good enough | 19:40 |
| sdague | it was an error message change, it will cycle back through again | 19:40 |
| auggy | yeah it's pretty trivial | 19:41 |
| fungi | auggy: we try to keep the ritual sacrifices around here to a minimum... apologies! | 19:41 |
| sdague | he | 19:41 |
| sdague | heh | 19:41 |
| sdague | fungi: ok, now that is done... node allocation question | 19:41 |
| sdague | it always seems like - gate-tempest-dsvm-neutron-full-ubuntu-xenial is the last thing to get allocated in the gate | 19:41 |
| sdague | is that because we don't have enough xenial nodes in the mix? | 19:42 |
| JayF | we had a test fail in the gate with POST_FAILURE and no logs: https://review.openstack.org/355290 | 19:42 |
| sdague | JayF: yep | 19:42 |
| sdague | log server filled up | 19:42 |
| JayF | obviously i can recheck, but it's a failure case I haven't seen before so didn't want to do it blindly | 19:42 |
| JayF | aha | 19:42 |
| jroll | JayF: 19:26:58 fungi | /dev/mapper/main-logs 12T 12T 0 100% /srv/static/logs | 19:42 |
| sdague | fixed now | 19:42 |
| jroll | it's dealt with | 19:42 |
| JayF | thanks! I'll recheck | 19:42 |
| greghaynes | cloudnull: question on https://review.openstack.org/#/c/356456/1 - I didnt follow all the detail but I remember mention of some of the defaults being distro specific? I'm wondering if this is a thing we want for ubuntu/debian distros/all distros | 19:42 |
| *** dimtruck is now known as zz_dimtruck | 19:43 | |
| jroll | so random question - do we do any sort of monitoring for these sorts of things? (or services in general?) would folks find that valuable, if some humans had time to work on that? | 19:43 |
| fungi | #status notice The volume for logs.openstack.org filled up rather suddenly, causing a number of jobs to fail with a POST_FAILURE result and no logs; we're manually expiring some logs now to buy breathing room, but any changes which hit that in the past few minutes will need to be rechecked and/or approved again | 19:43 |
| openstackstatus | fungi: sending notice | 19:43 |
| pabelanger | jroll: I've been meaning to get nagios going for us, but haven't found the time | 19:44 |
| fungi | jroll: it comes up periodically, nobody has had time | 19:44 |
| pabelanger | that | 19:44 |
| jroll | noted, thanks | 19:44 |
| jroll | pabelanger: dead set on nagios or open to other things? because familiarity, I'd assume? | 19:44 |
| fungi | it needs 1. people willing to design and implement, 2. people willing to monitor the output from it, and 3. people willing to tune it on an ongoing basis | 19:44 |
| JayF | If someone wanted to make time but do it hosted, i.e. Rackspace Cloud Monitoring, would that be acceptable? Or does the infra have to be owned/run by openstack | 19:44 |
| *** xarses_ has joined #openstack-infra | 19:44 | |
| -openstackstatus- NOTICE: The volume for logs.openstack.org filled up rather suddenly, causing a number of jobs to fail with a POST_FAILURE result and no logs; we're manually expiring some logs now to buy breathing room, but any changes which hit that in the past few minutes will need to be rechecked and/or approved again | 19:45 | |
| *** spzala has joined #openstack-infra | 19:45 | |
| fungi | JayF: i think that only covers #1 in the list, which is the easy part | 19:45 |
| jroll | fungi: does (2) need to be infra root, or just willing to incessantly ping infra root? | 19:45 |
| pabelanger | jroll: I had some puppet-nagios manifests that worked pretty well that used the nsca client. But I haven't touched that in years | 19:45 |
| fungi | jroll: #2 could be a dedicated mailing list, a dedicated irc channel, whatever populated with humans that can determine if a monitoring event is something that needs further investigation vs adjustment of the monitors | 19:46 |
| *** rbuzatu has joined #openstack-infra | 19:46 | |
| fungi | jroll: or even just people checking a web dashboard | 19:46 |
| cloudnull | greghaynes: I think we want that setting for all of the distros | 19:46 |
| jroll | fungi: *nod* | 19:46 |
| openstackstatus | fungi: finished sending notice | 19:46 |
| cloudnull | though tbh I've only tested things w/ cent and ubuntu, it may not even be needed in other distros (like if they've set that value to 1 or 0). | 19:47 |
| fungi | cloudnull: for that matter, it didn't seem to be causing any issues on xenial even, only trusty | 19:48 |
| greghaynes | cloudnull: oh, it was needed on centos? | 19:48 |
| fungi | so might be specific to the privacy addresses implementation in the trusty vintage kernel | 19:48 |
| *** jcoufal has quit IRC | 19:48 | |
| jroll | fungi: thanks for the info, I'll try to see if I can carve some time out (or nerd-snipe other humans that have some time) to work on this next cycle or so | 19:49 |
| cloudnull | fungi: thats a good point, later releases of distros may work +/- better w/ those settings out of the box | 19:49 |
| *** spzala has quit IRC | 19:49 | |
| fungi | for infra's purposes i'm fine disabling privacy addresses across the board, but for dib more generally that's not our call | 19:49 |
| *** spzala has joined #openstack-infra | 19:50 | |
| scottda | yolanda: Could you please have another look at https://review.openstack.org/#/c/330678/ ? You had already approved, but it needed a rebase. | 19:50 |
| fungi | i leave hard decisions like that up to greghaynes et al ;) | 19:50 |
| *** gyee has joined #openstack-infra | 19:50 | |
| cloudnull | greghaynes: no, that setting isn't specifically needed for cent7, in their cloud image the default is 0. | 19:50 |
| *** rbuzatu has quit IRC | 19:50 | |
| cloudnull | the file is simply ensuring that the options stay that way | 19:51 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: PoC of TripleO scenario experimental job https://review.openstack.org/356675 | 19:51 |
| fungi | sdague: on xenial node allocation, nodepool does try to figure out a proportional weighting of what to build based on demand but it's not perfect and may be guessing incorrectly. also i was seeing some jobs running on xenial nodes earlier today spontaneously requeue after being partway completed, so we might have something killing xenial we haven't had time to dig up details on | 19:51 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Remove unnecessary NodePoolBuilder thread https://review.openstack.org/356676 | 19:52 |
| cloudnull | i just made sure that cent had the same mechanism for "/etc/sysctl.d" | 19:52 |
| sdague | fungi: ok, is there a good way to query for that? | 19:52 |
| sdague | because I see it often enough, I was wondering | 19:52 |
| sdague | like top of gate | 19:52 |
| sdague | all the rest of the jobs were done, and it seemed to be just starting the xenial one | 19:52 |
| pabelanger | we also have a large about of ubuntu-xenail nodes fail to launch because of dns timeout in our ready-script | 19:53 |
| pabelanger | fix is incoming for that | 19:53 |
| EmilienM | any zuul expert could help me here please https://review.openstack.org/#/c/356675/ - I have 2 questions in the commit message | 19:53 |
| pabelanger | https://review.openstack.org/#/c/355695/ works around the dns failures | 19:53 |
| fungi | sdague: yeah, and i saw a few like that earlier go from partway complete to "queued" too but haven't had time to go hunting in zuul debug logs for the cause yet. we don't have anywhere public facing that tracks job aborts and requeues afaik | 19:54 |
| *** nwkarsten has quit IRC | 19:54 | |
| *** kzaitsev_mb has quit IRC | 19:54 | |
| *** nwkarsten has joined #openstack-infra | 19:55 | |
| pabelanger | sdague: fungi: a lot of xenail jobs are trying to run on osic-cloud1, which are failing to launch. That's the currently delay | 19:55 |
| jroll | EmilienM: for your question about experimental queues, yes that will work | 19:55 |
| EmilienM | jroll: oh wow, awesome | 19:55 |
| jroll | EmilienM: finding an example of repo/file match now | 19:55 |
| fungi | pabelanger: why would they run there if we set the quota in nodepool down to 0? | 19:55 |
| *** rfolco has joined #openstack-infra | 19:55 | |
| jroll | EmilienM: I should say I'm 95% sure it will work :) | 19:55 |
| sdague | pabelanger: really, I thought we removed them this morning | 19:55 |
| pabelanger | fungi: we only set ubuntu-trusty to 0 | 19:55 |
| EmilienM | jroll: ok | 19:55 |
| sdague | oh, right, sorry | 19:55 |
| pabelanger | I though | 19:55 |
| sdague | pabelanger: you are right | 19:56 |
| *** nwkarste_ has joined #openstack-infra | 19:56 | |
| pabelanger | err: removed | 19:56 |
| sdague | I was mixing up things | 19:56 |
| fungi | pabelanger: oh, so we're running a bunch of additional xenial jobs in osic i guess | 19:56 |
| pabelanger | yes, it is the only cloud with capacity right now | 19:56 |
| sdague | fungi: right, the description yesterday was that it was a trusty only issue | 19:56 |
| *** kcobb has quit IRC | 19:56 | |
| cloudnull | yea its been all xenial all day. :) | 19:56 |
| pabelanger | and because urandom issue, launchs usually fail | 19:56 |
| sdague | so this morning we just turned off trusty for osic | 19:56 |
| sdague | because that seemed like we wouldn't loose capacity too badly | 19:56 |
| jroll | EmilienM: so this is skipping based on repo, I'm wondering if there's a 'run-if' similar to 'skip-if' https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1199 | 19:57 |
| sdague | and would prevent the known fail | 19:57 |
| jroll | seems like something that could exist | 19:57 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic show how choppy we are in osic-cloud1 right now | 19:57 |
| mordred | pabelanger: both those changes look good to me | 19:57 |
| pabelanger | which is our ubuntu-xenail failures | 19:57 |
| sdague | pabelanger: does that get better with the ipv6 fix? | 19:57 |
| sdague | or unrelated | 19:57 |
| sdague | ? | 19:57 |
| EmilienM | jroll: I checked in zuul and I don't think we can specific files + project | 19:58 |
| pabelanger | no, this is a kernel issue I think with urandom | 19:58 |
| pabelanger | causing unbound to take up to 60sec to start | 19:58 |
| pabelanger | but nodepool has already ssh'd into the node and done host git.openstack.org, which fails | 19:58 |
| pabelanger | deletes node, repeats | 19:58 |
| *** nwkarsten has quit IRC | 19:59 | |
| *** pvaneck has quit IRC | 20:00 | |
| fungi | 2016-08-17 19:25:25,520 INFO zuul.Gearman: Build <gear.Job 0x7fc0fae11890 handle: H:127.0.0.1:1193623 name: build:gate-tempest-dsvm-neutron-full-ubuntu-xenial unique: 90933c78945a45eb81d65e3b4d307f80> complete, result None | 20:00 |
| *** pvaneck has joined #openstack-infra | 20:00 | |
| fungi | sdague: that's the predecessor to the job running on 356211,1 right now | 20:00 |
| fungi | the job "completed" with a result of None | 20:01 |
| sdague | fungi: so it is xenial recycling? | 20:01 |
| sdague | so probably pabelanger's thing | 20:01 |
| fungi | it's something but not sure what yet | 20:01 |
| fungi | well, it ran for some half hour before it completed | 20:01 |
| fungi | i'm tracking it back to launcher logs now | 20:02 |
| jroll | EmilienM: meh, I think you're right... you could probably hack it to have slightly different names for tripleo vs puppet (which both reference the same job) | 20:02 |
| *** rwsu has joined #openstack-infra | 20:02 | |
| openstackgerrit | Merged openstack-infra/nodepool: Clarify code in nodepool/cmd/builder.py https://review.openstack.org/356504 | 20:02 |
| EmilienM | jroll: thanks for looking too :-) | 20:02 |
| jroll | EmilienM: sure, not a problem :) | 20:02 |
| EmilienM | jroll: sounds like I found an rfe ;-) | 20:02 |
| jroll | EmilienM: if you're *really* bored, I'm sure it'd be ... yes! | 20:02 |
| fungi | sdague: i have a job start event for that zuul id of 18:53:07 utc | 20:02 |
| jroll | heh | 20:02 |
| sdague | fungi: huh, weird | 20:03 |
| fungi | it was launched from zl04 so i'm digging around in its debug log for hints | 20:03 |
| *** Sukhdev has quit IRC | 20:06 | |
| fungi | mordred: jeblair: pabelanger: Shrews: what are the odds that one of the conditions causing spontaneous POST_FAILURE results at job completion has recently switched to just reenqueuing jobs instead? was that a mostly xenial behavior by any chance? | 20:08 |
| fungi | jeblair: ^ ? | 20:08 |
| fungi | er, i already had you on there, sorry | 20:08 |
| *** devkulkarni has quit IRC | 20:09 | |
| Shrews | fungi: iirc, there was a very recent change to zuul to auto-requeue | 20:09 |
| cloudnull | pabelanger fungi i wonder if we could seed entropy when the instance boots using something like user-data and a simple-ish command like ``dd if=/dev/urandom of=/dev/random count=32 bs=256b``? | 20:09 |
| cloudnull | idk if that'd even work but it makes sense in my brain it would. | 20:09 |
| fungi | cloudnull: it's unfortunately not that simple, but jeblair is working on a similarish solution | 20:10 |
| Shrews | fungi: but i'm not qualified enough to say more than that unuseful statement | 20:10 |
| cloudnull | ok | 20:10 |
| clarkb | fungi: is it possible the log volume filling is related to the restarting jobs? | 20:10 |
| *** amitgandhinz has quit IRC | 20:10 | |
| pabelanger | fungi: let me check, but I don't think we requeue on POST_FAILURE | 20:10 |
| fungi | clarkb: maybe? seems odd that would mostly impact xenial though | 20:10 |
| *** amitgandhinz has joined #openstack-infra | 20:10 | |
| fungi | here's what i've got for the job run in question from the launcher: http://paste.openstack.org/show/559092/ | 20:11 |
| clarkb | as for logs filling sdague was talking about some huge keystone logs | 20:11 |
| fungi | "status": null | 20:11 |
| clarkb | the log issue could be related to that possibly | 20:11 |
| *** coolsvap has quit IRC | 20:11 | |
| fungi | clarkb: yeah, that's only over the last couple days though, while the upturn in log volume took us from 6tb to 12tb in 2 months | 20:12 |
| sdague | clarkb: that would be really recent | 20:12 |
| clarkb | http://logs.openstack.org/11/356211/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/90933c7 is a 404 | 20:12 |
| sdague | this has been going on for longer | 20:12 |
| *** xarses_ has quit IRC | 20:12 | |
| fungi | clarkb: yep, it didn't upload logs, that much i can tell | 20:12 |
| clarkb | so the status null didn't manage to log anything (either due to not trying to or due to full lgo servers) | 20:12 |
| *** xarses_ has joined #openstack-infra | 20:12 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:13 | |
| pabelanger | I thought we only requeued on -9 (request abort) or 3 (AnsibleHostUnreachable) | 20:13 |
| jeblair | oh hi i am back from lunch | 20:14 |
| pabelanger | fungi: 2016-08-17 19:25:25,515 DEBUG zuul.NodeWorker.ubuntu-xenial-osic-cloud1-3569008: Ansible exit code: 3 | 20:14 |
| pabelanger | so, we requeue because network failure | 20:14 |
| fungi | pabelanger: okay, so we may still be seeing some network connectivity failures for xenial in osic, and we're requeueing those i guess? | 20:15 |
| *** rhallisey_ has joined #openstack-infra | 20:15 | |
| pabelanger | fungi: that would make sense | 20:15 |
| pabelanger | I have ubuntu-trusty rebuilding | 20:15 |
| pabelanger | I can check ubuntu-xenail too | 20:15 |
| fungi | that would also explain why it's impacting xenial heavily, since we temporarily turned off trusty in osic and their quota accounts for 1/3 of our aggregate | 20:16 |
| clarkb | fungi: good point | 20:16 |
| fungi | which probably means a majority (or close to it) of our xenial jobs are running there | 20:16 |
| pabelanger | yup | 20:17 |
| clarkb | this sort of thing is similar to what I saw with the zuul lauincher restart script from puppetmaster | 20:17 |
| clarkb | the polls don't always work which causes your long lived ansible playbook to abort halfway through executing | 20:17 |
| fungi | oh, perhaps this could also be blamed on v6 privacy addresses then | 20:18 |
| *** ayoung has joined #openstack-infra | 20:18 | |
| fungi | if it somehow failed to respond sourced from the correct address or something | 20:18 |
| clarkb | maybe? the change to did should affect xenial too right? | 20:18 |
| clarkb | so if we can get new images built and uploaded we can at least check that? | 20:18 |
| fungi | yep | 20:18 |
| clarkb | s/did/dib/ | 20:19 |
| *** dprince has quit IRC | 20:19 | |
| *** rhallisey_ has quit IRC | 20:20 | |
| jeblair | fungi, pabelanger: i have completed my program, essay, and lunch, and am ready to upload two of the three now. where do we want it? system-config puppet, or project-config element, or ...? | 20:20 |
| *** e0ne has joined #openstack-infra | 20:20 | |
| mordred | jeblair: are you going to upload your lunch? | 20:20 |
| mordred | jeblair: I thnk that influences where I think it should go | 20:20 |
| cloudnull | Fungi I'd bet that the privacy settings g are having a larger impact than just trusty. | 20:21 |
| *** dizquierdo has joined #openstack-infra | 20:21 | |
| clarkb | jeblair: maybe start with project-config element to reduce further image builds on puppet? then if we need it on our other servers move into system-cofnig? | 20:21 |
| mordred | jeblair: yah. I agree with clarkb | 20:21 |
| clarkb | *image build deps on puppet | 20:21 |
| clarkb | thats a mouthful and I fail at typing it | 20:21 |
| fungi | jeblair: hard to say without knowing more. does it need a build step or did ctypes do teh trick? | 20:21 |
| jeblair | fungi: ctypes works | 20:21 |
| fungi | yeah, project-config seems reasonable then | 20:22 |
| jeblair | okay, if i have an element called "initialize-urandom" where in the element tree would the script that we're installing go? | 20:22 |
| mordred | jeblair: is there any chance this program could be more broadly useful to other people? | 20:22 |
| fungi | i don't see much point in puppetizing if this is really only targeting single-use job nodes | 20:22 |
| fungi | jeblair: install.d | 20:22 |
| fungi | (i think) | 20:22 |
| fungi | dib still seems like voodoo to me some days | 20:23 |
| mordred | we have some examples of installing scripts ... | 20:23 |
| mordred | one sec | 20:23 |
| *** Hal1 has quit IRC | 20:23 | |
| jeblair | mordred: i hope it's not generally useful; the thing it's fixing has already been removed from the kernel | 20:23 |
| mordred | cool | 20:23 |
| jeblair | i think we're only going to need it for the lifetime of xenial nodes | 20:23 |
| *** javeriak has joined #openstack-infra | 20:23 | |
| mordred | jeblair: install.d is where you'd put the script that installs the file | 20:23 |
| fungi | unless ubuntu decides to backport newer kernels soon | 20:24 |
| mordred | jeblair: in that script you'll do a copy ... but I'm trying to find an example of that copy to hand you | 20:24 |
| cloudnull | When I tested yesterday I built several vms on every single compute node to test both trusty and xenial and i couldnt produce the issues using vanilla images; both distro provided cloud images have the privacy settings disabled by default. | 20:24 |
| fungi | cloudnull: bizarre. any idea where it's coming from in our dib minimal builds then? | 20:24 |
| cloudnull | The privacy setting? | 20:25 |
| fungi | cloudnull: yeah | 20:25 |
| clarkb | fungi: I think ubuntu's kernels default to 2, then ubuntu's cloud image builds slap a file in to undo that | 20:25 |
| mordred | ah | 20:25 |
| cloudnull | It's the system default. | 20:25 |
| fungi | wondering how they're getting turned on if the kernel default is 0 | 20:25 |
| mordred | jeblair: the install-static element allows you to install static files into the image | 20:25 |
| cloudnull | Kernel default in Ubuntu is 2 | 20:25 |
| jeblair | mordred: where in the hierarchy should the file go? | 20:26 |
| cloudnull | Ha! What Clarkb said. | 20:26 |
| fungi | oh, so ubuntu compiles their kernel with a nondefault setting for that and then overrides it back to the default in configuration? that's... bonkers? | 20:26 |
| mordred | jeblair: so put the file in static/usr/local/bin/myscript.py | 20:26 |
| clarkb | right so kernel kernel default is 0, ubuntu kernel default is 2, ubuntu cloud images then undo the ubuntu kernel default to match the kernel kernel | 20:26 |
| mordred | jeblair: then add install-static to the element depends | 20:26 |
| clarkb | fungi: YUP | 20:26 |
| jeblair | initialize-urandom/static/usr/local/bin/myscript.py ? | 20:26 |
| mordred | jeblair: yah | 20:26 |
| *** javeriak_ has quit IRC | 20:26 | |
| sdague | fungi: yeh, I ran across stuff like this doing a similar kind of project a decade ago | 20:26 |
| cloudnull | Fungi. I scratched my head at that for hours last night ;-) | 20:27 |
| mordred | jeblair: then put "install-static" into initialize-urandom/element-deps | 20:27 |
| jeblair | mordred: i'm going to leave that for pabelanger or someone else | 20:27 |
| sdague | it is amazing how many edge cases the distros just fix with a post install config tweak that's not encoded in packages | 20:27 |
| jeblair | i just want to get the script up first | 20:27 |
| mordred | kk | 20:27 |
| *** javeriak has quit IRC | 20:28 | |
| fungi | sdague: but... they're intentionally configuring the behavior at kernel compile-time and then unconfiguring it back to the upstream default at runtime. that's what gets me | 20:28 |
| *** kzaitsev_mb has quit IRC | 20:29 | |
| odyssey4me | fungi perhaps the package comes from debian, not ubuntu - then ubuntu tweaks it? | 20:29 |
| sdague | fungi: yeh, honestly it could be something like 2 different teams responsible for kernel package vs. system install | 20:29 |
| jeblair | well, much easier to turn the feature on if you don't have to recompile the kernel | 20:29 |
| odyssey4me | pure speculaiton of course | 20:29 |
| sdague | jeblair: yeh, that too | 20:29 |
| odyssey4me | there is also that | 20:29 |
| fungi | sdague: i'm guessing this goes back to ubuntu really being mostly a desktop distro with servers as a bolted-on use case, so they're turning on privacy addressing in their kernel package for the benefit of desktops and then unconfiguring it in server installs | 20:30 |
| mordred | fungi: from what I can tell from the bug, there is a disagreement between two different camps on what the "right" behavior is | 20:30 |
| *** _ari_ has quit IRC | 20:30 | |
| mordred | ++ | 20:30 |
| mordred | exactly | 20:30 |
| *** amitgandhinz has quit IRC | 20:30 | |
| mordred | the desktop people vehemently disagree that having this on by default is broken | 20:30 |
| sdague | right, that could be as well | 20:31 |
| *** amitgandhinz has joined #openstack-infra | 20:31 | |
| fungi | though there is also the camp of (in my opinion paranoid) admins who want privacy addressing for servers to make it harder to enumerate their ipv6 network addresses or guess what kind of network hardware they have | 20:32 |
| mordred | sure. I just think that's an opt in | 20:33 |
| fungi | i'm not keen on pandering to the security-by-obscurity camp personally | 20:33 |
| dmsimard | o/ Is it a requirement for openstack-infra nodepool "sponsors" to provide nested virt ? I just realized that devstack tests with virt_type = qemu (as opposed to virt_type = kvm) | 20:33 |
| sdague | fungi: and there is the camp that finds this destroys their network - https://blog.bimajority.org/2014/09/05/the-network-nightmare-that-ate-my-week/ | 20:33 |
| mordred | dmsimard: we do not make use of nested virt anywhere | 20:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Nodepool elements: Add a script to initialize urandom https://review.openstack.org/356697 | 20:34 |
| fungi | sdague: yep! | 20:34 |
| fungi | dmsimard: most of our donors do not provide nested virtualization acceleration | 20:34 |
| dmsimard | mordred: We've been testing Packstack in upstream gate with virt_type = kvm for quite a while already | 20:34 |
| mordred | dmsimard: one of our large providers runs on xen | 20:34 |
| jeblair | mordred, fungi, clarkb, cloudnull, bkero, pabelanger, greghaynes: please see https://review.openstack.org/356697 | 20:34 |
| dmsimard | ... interesting | 20:34 |
| *** pfallenop has quit IRC | 20:35 | |
| sdague | dmsimard: also, nested kvm randomly crashes | 20:35 |
| dmsimard | We have a patchset to switch puppet-openstack-integration from qemu to kvm, mostly out of curiosity | 20:35 |
| *** Jokke_ has quit IRC | 20:35 | |
| sdague | enough so that with gate volume, it is noticable | 20:35 |
| * cloudnull looking. | 20:35 | |
| sdague | so it was specifically disabled even on providers that allow it | 20:36 |
| *** tphummel has quit IRC | 20:36 | |
| *** nwkarste_ has quit IRC | 20:36 | |
| *** Jokke_ has joined #openstack-infra | 20:37 | |
| *** nwkarsten has joined #openstack-infra | 20:37 | |
| fungi | sdague: you probably want to remind mikal of that. he was pushing pretty hard in austin for getting nested virt going again in jobs | 20:38 |
| *** nwkarste_ has joined #openstack-infra | 20:38 | |
| sdague | fungi: ok, I thought I had reminded him last time we drank beers | 20:38 |
| fungi | oh, that's probably more recent than austin then | 20:38 |
| jeblair | if he drank too many beers you may need to remind again | 20:39 |
| sdague | jeblair: that is also probable | 20:39 |
| *** armax has joined #openstack-infra | 20:39 | |
| fungi | i knew we had disabled it for a variety of reasons, but didn't have the wherewithal to enumerate them | 20:39 |
| sdague | ground hog day of beers | 20:39 |
| bkero | jeblair: I can't help but wonder if this code shouldn't be in systemd | 20:40 |
| odyssey4me | dmsimard we do some detection to try and figure out what's available, then go with it: https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_virt_detect.yml | 20:40 |
| *** rcernin has quit IRC | 20:40 | |
| dmsimard | fungi, jeblair, mordred: thanks for your answers. Turns out Packstack does fallback to qemu when it detects it's running in a VM. I wrongly assumed it was using KVM. | 20:41 |
| *** pfallenop has joined #openstack-infra | 20:41 | |
| dmsimard | So I was confused as to how it worked | 20:41 |
| fungi | bkero: all code everywhere will be in systemd given sufficient time | 20:41 |
| *** nwkarsten has quit IRC | 20:41 | |
| dmsimard | fungi: lol. | 20:42 |
| bkero | fungi: Yeah, I just mean that systemd has code to do this, but it doesn't call the credit_entropy_pool() function like it should. | 20:42 |
| bkero | So it's just broken and useless. | 20:42 |
| fungi | but, you know, it has to start out somewhere ;) | 20:42 |
| jeblair | bkero: i don't think so, actually. i believe that it's intentional that the randomness supplied by systemd is not credited with entropy because it's not assured that it really represents entropy | 20:42 |
| bkero | jeblair: Why do it at all then? | 20:42 |
| mordred | jeblair: change looks great | 20:43 |
| jeblair | bkero: voodoo? it's better than nothing? it does add data to the pool. | 20:43 |
| fungi | bkero: also, with it no longer being a problem relevant to newer kernels, they might just not care unless there's a rhel release with an affected kernel | 20:43 |
| jeblair | that too | 20:43 |
| *** roxanaghe has quit IRC | 20:44 | |
| bkero | It adds data to the pool, but the data is useless because they're not calling the ioctl | 20:44 |
| jeblair | this is a pretty narrow window, which has already closed. we just got stuck inside. | 20:44 |
| fungi | systemd implementing a specific fix for a kernel only found on ubuntu 16.04 is... dubious | 20:44 |
| fungi | as they're very much red-hat-centric | 20:44 |
| SamYaple | fungi: what. i have clearly missed something | 20:44 |
| bkero | 16.04, 14.04, and el7 | 20:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Update osic-cloud1 credential format https://review.openstack.org/356702 | 20:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add credentials for osic-cloud8 https://review.openstack.org/356703 | 20:45 |
| SamYaple | did this happen or is it a purposed patch? | 20:45 |
| *** kzaitsev_mb has joined #openstack-infra | 20:45 | |
| fungi | bkero: the problem behavior crops up starting with linux 3.17 though | 20:45 |
| pabelanger | so, osic-cloud8 is ready for us! | 20:45 |
| jeblair | bkero: it's not useless -- it is actually used in generating random numbers | 20:45 |
| pabelanger | I haven't update hiera yet with the info, but wanted to get the patches online first | 20:45 |
| pabelanger | Hope to use cloud-launcher to setup openstack projects too | 20:46 |
| fungi | bkero: rhel 7 has too old of a kernel to see it, as does debian jessie... the next rhel, debian and ubuntu lts releases will almost certainly have kernels with the newer entropy management that obsoletes all of this anyway | 20:46 |
| *** tphummel has joined #openstack-infra | 20:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add glue to get initialize-urandom installed https://review.openstack.org/356706 | 20:46 |
| mordred | jeblair: ^^ | 20:46 |
| jeblair | mordred: thanks! | 20:47 |
| *** ansmith has quit IRC | 20:47 | |
| jeblair | pabelanger: you want to build on 356697 and 356706 to add the systemd bits? | 20:47 |
| pabelanger | jeblair: sure. trying to figure out our git repo failures now | 20:48 |
| mordred | pabelanger: or feel free to just hijack 356706 if that makes sense | 20:48 |
| clarkb | dmsimard: sdague not only do the instances crash they disapear with no way to debug them | 20:48 |
| jeblair | pabelanger: also, i'm finished using your test instance, if you want to use it for this | 20:48 |
| pabelanger | jeblair: okay | 20:48 |
| pabelanger | mordred: will do | 20:48 |
| pabelanger | fungi: see my comments about osic-cloud8 when free | 20:49 |
| *** jheroux has quit IRC | 20:49 | |
| jeblair | pabelanger: we have git repo failures? | 20:49 |
| fungi | pabelanger: yep, plan to review as soon as i finish reading through 356697 | 20:49 |
| *** jkilpatr has quit IRC | 20:49 | |
| pabelanger | jeblair: empty repo: http://git.openstack.org/cgit/openstack/deb-websockify | 20:50 |
| pabelanger | we did a large import this morning | 20:50 |
| fungi | jeblair: there were some hundreds of debian packaging repos approved earlier today. i don't know if anyone has had time to go back and confirm they all imported correctly | 20:50 |
| *** dizquierdo has quit IRC | 20:51 | |
| pabelanger | looks to exist in review.o.o | 20:52 |
| pabelanger | maybe not replicated? | 20:52 |
| fungi | pabelanger: or maybe missing a master branch? | 20:52 |
| pabelanger | Oh, maybe | 20:52 |
| fungi | though i thought we added a repo checker to confirm proposed imports included a master branch, so probably not | 20:53 |
| *** Hal has joined #openstack-infra | 20:53 | |
| pabelanger | ya, import didn't work | 20:54 |
| pabelanger | just an empty repo | 20:54 |
| pabelanger | checking manage-projects | 20:55 |
| fungi | there is a manage-projects log (somewhere in /var) you can skim | 20:55 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Deploy minimal services in multinode job https://review.openstack.org/355097 | 20:56 |
| *** raildo has quit IRC | 20:57 | |
| *** acoles_ is now known as acoles | 20:58 | |
| *** hyakuhei has joined #openstack-infra | 20:58 | |
| *** edmondsw has joined #openstack-infra | 20:58 | |
| pabelanger | fungi: http://paste.openstack.org/show/559098/ | 21:00 |
| *** ldnunes has quit IRC | 21:00 | |
| pabelanger | nothing popping out as a problem | 21:00 |
| *** e0ne has quit IRC | 21:00 | |
| clarkb | pabelanger: fungi the other thing to check si the replication queue in gerrit? perhaps things got slow? | 21:01 |
| *** pbourke has quit IRC | 21:01 | |
| fungi | fatal: bad default revision 'HEAD' (on disk in gerrit when trying to git log) | 21:02 |
| *** edtubill has quit IRC | 21:02 | |
| fungi | HEAD contains ref: refs/heads/master | 21:03 |
| clarkb | there are hundreds of retries to github but don't see any problems with git.o.o | 21:03 |
| *** pbourke has joined #openstack-infra | 21:03 | |
| *** gouthamr has quit IRC | 21:04 | |
| *** annegentle has joined #openstack-infra | 21:04 | |
| *** rhallisey has quit IRC | 21:04 | |
| fungi | seems there's no master branch in the bare repo on disk | 21:04 |
| clarkb | debian/newton is the default branch | 21:04 |
| clarkb | so that is likely the issue | 21:05 |
| fungi | when i clone from https://anonscm.debian.org/git/openstack/deb-websockify.git i do see remote branches for debian/newton and master | 21:05 |
| pabelanger | yes, I see that in /opt/lib/jeepyb/openstack/deb-websockify too | 21:05 |
| clarkb | ya its not sufficient to have a master branch. It also needs to be HEAD | 21:05 |
| clarkb | because HEAD isn' | 21:05 |
| clarkb | t updated when you push | 21:05 |
| pabelanger | clarkb: okay, zigo can just fix that upstream right? | 21:05 |
| fungi | well, we set HEAD to master for gerrit imports | 21:05 |
| fungi | but we didn't seem to clone master (or any branches at all) | 21:06 |
| *** sdague has quit IRC | 21:06 | |
| fungi | so i'm unconvinced that's the issue | 21:06 |
| clarkb | fungi: you are saying push ssh://review.openstack.org:29418/openstack/deb-websockify +refs/copy/heads/*:refs/heads/* didn't have the desired effect? | 21:06 |
| fungi | yeah, i'm seeing if the temporary copy jeepyb makes sheds any more light on this | 21:07 |
| *** zz_dimtruck is now known as dimtruck | 21:07 | |
| *** camunoz has quit IRC | 21:07 | |
| *** psilvad has quit IRC | 21:08 | |
| *** gordc has quit IRC | 21:08 | |
| *** narayrak has quit IRC | 21:08 | |
| openstackgerrit | Merged openstack/os-testr: Add support for Python versions https://review.openstack.org/355730 | 21:08 |
| clarkb | locally git fetch origin +refs/heads/*:refs/copy/heads/* populates the refs as I expect | 21:09 |
| fungi | looks like /opt/lib/jeepyb/openstack/deb-websockify has debian/newton and master | 21:09 |
| *** annegentle has quit IRC | 21:10 | |
| fungi | but comparing to, e.g., deb-keystone it's missing remotes/origin/debian/newton and remotes/origin/master | 21:10 |
| *** ldnunes has joined #openstack-infra | 21:11 | |
| *** edtubill has joined #openstack-infra | 21:11 | |
| fungi | also /opt/lib/jeepyb/openstack/deb-websockify/.git/HEAD correctly has ref: refs/heads/master | 21:12 |
| *** fifieldt has quit IRC | 21:13 | |
| fungi | [2016-08-17 16:31:54,827] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user openstack-project-creator account 6199) during git-receive-pack '/openstack/deb-websockify' | 21:13 |
| clarkb | huh | 21:13 |
| hyakuhei | fungi do you know if I need some special privileges to be able to upload images to mediawiki? I'm working on updating the Security Note documentation | 21:14 |
| fungi | something about a problem reading the pack stream, per the backtrace | 21:14 |
| fungi | hyakuhei: we disabled image uploads because the confirmedit captcha extension didn't enforce a captcha for file uploads | 21:15 |
| *** rfolco has quit IRC | 21:15 | |
| fungi | so we were overrun with spam image uploads | 21:15 |
| hyakuhei | I figured it might be something like that | 21:15 |
| *** gyee_ has joined #openstack-infra | 21:15 | |
| *** thiagop has quit IRC | 21:15 | |
| fungi | hyakuhei: we want to try turning it back on and seeing if the upgraded version does | 21:15 |
| fungi | hyakuhei: mind helping me test that? | 21:16 |
| hyakuhei | Happy to | 21:16 |
| fungi | hyakuhei: just a sec and i'll let you know when we're ready | 21:16 |
| *** jkilpatr has joined #openstack-infra | 21:16 | |
| *** ldnunes has quit IRC | 21:17 | |
| *** spzala has quit IRC | 21:17 | |
| clarkb | fungi: so that all runs on localhost... hard to blame a network error in that instance though I guess bit flips or similar nastyness could still happen (yay memory) | 21:17 |
| clarkb | I wonder if we have somehow tripped over a bug in gerrit | 21:17 |
| clarkb | or jgit | 21:17 |
| fungi | clarkb: still easy to blame java | 21:17 |
| hyakuhei | Cool, just ping me on here | 21:17 |
| *** spzala has joined #openstack-infra | 21:18 | |
| fungi | hyakuhei: okay, i've temporarily reverted https://review.openstack.org/345100 on the server, so give it a try | 21:19 |
| *** gyee has quit IRC | 21:19 | |
| hyakuhei | Righto, standby! | 21:19 |
| inc0 | fungi, pabelanger so, you've got your toys;) let us know how they behave | 21:20 |
| *** kgiusti has left #openstack-infra | 21:20 | |
| pabelanger | inc0: fungi: Thanks, need to run some errands tonight, but should have an hour or so to launch mirror.regionone.osic-cloud8.openstack.org when I get back | 21:21 |
| *** spzala has quit IRC | 21:22 | |
| hyakuhei | Seems to work fungi :) | 21:22 |
| inc0 | cool, feel free to ping me, mrhillsman or raddaoui | 21:22 |
| pabelanger | fungi: mordred: we should also land 356637 to fix multinode jobs on osic-cloud1 too | 21:23 |
| fungi | hyakuhei: did you get prompted with a captcha? | 21:23 |
| mrhillsman | would be great to work at some point to get all ipv6 working :) wonder for how many ip allocation is a bottleneck to offering up more resources | 21:24 |
| hyakuhei | one second. | 21:24 |
| *** psachin has joined #openstack-infra | 21:24 | |
| *** kbaegis has quit IRC | 21:24 | |
| hyakuhei | Should I get a captcha challenge for every upload? I'm pretty sure I'd already done one on the wiki page, don't think the upload prompt itself challenged me for one | 21:24 |
| *** fifieldt has joined #openstack-infra | 21:25 | |
| clarkb | jeblair: did you consider using GRND_NONBLOCK rather than having two threads? | 21:25 |
| fungi | hyakuhei: i'm not completely sure, to be honest | 21:25 |
| fungi | hyakuhei: i'll play around with it some as well | 21:25 |
| dmsimard | clarkb: for the record virt_type = kvm works on OSIC nodes :P | 21:25 |
| *** mdrabe has quit IRC | 21:25 | |
| fungi | hyakuhei: though i have to deauthenticate and then sign back in with my unprivileged peon account to confirm, so it'll take me a bit | 21:25 |
| clarkb | dmsimard: and ovh and internap I think. Problem is you will crash your tests and put them in a completely non debuggable state by doing so | 21:25 |
| *** mdrabe has joined #openstack-infra | 21:26 | |
| cloudnull | we have nested virt enabled lir | 21:26 |
| jeblair | clarkb: oh, invoking it and checking it for EAGAIN after each iteration through the loop? no i did not. :) | 21:26 |
| clarkb | jeblair: ya | 21:26 |
| dmsimard | clarkb: yeah fair | 21:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Nodepool elements: Add a script to initialize urandom https://review.openstack.org/356697 | 21:27 |
| jeblair | better englished ^ | 21:27 |
| jeblair | clarkb: i think that would work. do you think it's worth changing? | 21:27 |
| fungi | pabelanger: on 356702 did you copy the hiera keys? | 21:28 |
| clarkb | jeblair: probably not. I am finding the chewy part of the code to be syscall rather than thread related | 21:28 |
| *** matt-borland has quit IRC | 21:28 | |
| clarkb | jeblair: I think it would reduce the code a bit though if you wanted to make it more concise | 21:28 |
| dmsimard | clarkb: perhaps providers with nested virt enabled could be labelled as such so that projects that wants to make use of it could | 21:29 |
| dmsimard | since it's not an official nodepool provider requirement | 21:29 |
| jeblair | clarkb: yeah, i mean, it might keep us from sending like an extra chunk of data. but this is probably good enough. | 21:29 |
| *** pradk has quit IRC | 21:29 | |
| clarkb | dmsimard: you can just check /proc/cpuinfo | 21:30 |
| clarkb | dmsimard: for the vmx flag ? I think maybe | 21:30 |
| dmsimard | I never realized it wasn't a requirement, we're testing OpenStack on OpenStack clouds, I guess I assumed it would be a requirement :/ | 21:30 |
| dmsimard | clarkb: yeah but then what, fail the job if vmx isn't there ? | 21:30 |
| dmsimard | We could schedule the jobs on providers that are labelled as such | 21:30 |
| fungi | pabelanger: and what's with the todo on 356703? and that port number... | 21:30 |
| clarkb | dmsimard: oh no I don't think we should do that | 21:31 |
| clarkb | dmsimard: that isn't a feature we want to expose because its not a reqwuirement and it isn't reliable | 21:31 |
| dmsimard | clarkb: you're completely comfortable with testing only one hypervisor upstream ? | 21:31 |
| clarkb | dmsimard: we have no reliable way to test anything else | 21:32 |
| fungi | dmsimard: we test libvirt | 21:32 |
| clarkb | kvm require features that don't work and xen requires complicated image management | 21:32 |
| fungi | it's up to libvirt to test kvm or qemu | 21:32 |
| *** kbaegis has joined #openstack-infra | 21:32 | |
| clarkb | also ^ | 21:32 |
| pabelanger | fungi: I have yet to copy them, I can do that now. If naming schema is fine | 21:32 |
| jeblair | dmsimard: we treat providers as failable components in a fault-tolerant system. they fail quite often are added and removed at an increasingly brisk pace. | 21:32 |
| *** devkulkarni has joined #openstack-infra | 21:32 | |
| fungi | pabelanger: i'm fine with the naming schema, less sure about the second change though | 21:32 |
| pabelanger | fungi: okay, let me do first, then review 2nd | 21:33 |
| jeblair | pabelanger: if i can borrow that vm for a few more minutes, i'm going to poke at clarkb's idea | 21:33 |
| *** Sukhdev has joined #openstack-infra | 21:33 | |
| *** annegentle has joined #openstack-infra | 21:35 | |
| *** roxanaghe has joined #openstack-infra | 21:35 | |
| *** Hal has quit IRC | 21:35 | |
| *** kbaegis1 has joined #openstack-infra | 21:36 | |
| pabelanger | jeblair: sure | 21:36 |
| *** kbaegis has quit IRC | 21:37 | |
| clarkb | jeblair: and is entropy_count bits not bytes? | 21:37 |
| jeblair | clarkb: correct | 21:37 |
| clarkb | that explains the *8 thanks | 21:38 |
| jeblair | (that's based on haveged's own assertion of the entropy of the data it sends) | 21:38 |
| fungi | which mirrors the kernel's tracking of entropy as bits | 21:39 |
| fungi | the pool sizes and states are represented in bitvalues | 21:39 |
| pabelanger | inc0: fungi: So, I think cloud8.osic.org is going to be the DNS. We could wait for that? Oh, and get SSL too | 21:39 |
| jeblair | fungi: yeah -- though i mean specificy saying that one byte of data contains 8 bits of entropy is an assertion from haveged -- it could send a byte of data that contains 4 bits of entropy. i guess. but it doesn't. :) | 21:40 |
| clarkb | jeblair: ya this seems reasonable sane (tahnks for the long docstring). My only suggestion is using the nonblocking flag and checking eagain may be simpler code to follow because it can be done with one thread | 21:40 |
| clarkb | btu I don't think that is required | 21:40 |
| jeblair | clarkb: yeah, i'll try it out. | 21:40 |
| fungi | pabelanger: i just recall some complication with the first osic environment we turned up around ssl/tls and dns, so don't want to jump the gun if we aren't actually able to use plan http there | 21:41 |
| fungi | jeblair: yep, got it. haveged is doing whitening/compression to make sure it supplies 1:1 entropy bit per data bit | 21:41 |
| clarkb | jeblair: that may also work into addressing cloudnull's comment | 21:42 |
| pabelanger | fungi: okay, have to step out for errands, but once I get back I can try using HTTP. However, maybe we should just wait for https / dns.... _shurgs_ | 21:42 |
| clarkb | fungi: pabelanger complications with cloud1 were we couldn't trust the self signed cert even if we wanted to because the entries in keystone catalog all used the ip addr so the names in the cert didn't match | 21:42 |
| fungi | ahh, right | 21:43 |
| clarkb | fungi: pabelanger: so we just waited for them to get a proper cert and up to data catalog | 21:43 |
| pabelanger | right | 21:43 |
| fungi | there is catalog madness associated with raw ip addresses | 21:43 |
| pabelanger | I remember | 21:43 |
| clarkb | in theory we could trust a self signed cert as long as the catalog matched the name(s) in the cert | 21:43 |
| pabelanger | IIRC, osic-cloud1 is using letsencrypt | 21:44 |
| cloudnull | mrhillsman inc0 raddaoui -- see pabelanger, fungi, and clarkb questions. RE cloud8. | 21:44 |
| cloudnull | pabelanger: ++ we're using letsencrypt | 21:44 |
| cloudnull | which has kicked us at least once ... | 21:44 |
| pabelanger | Expires On Wednesday, August 24, 2016 at 2:27:00 PM | 21:44 |
| pabelanger | coming up! | 21:44 |
| fungi | pabelanger: though 356703 still needs the port number in the auth_url for nodepool corrected i think, per my inline comment | 21:44 |
| inc0 | pabelanger, sorry, I missed that, you want cloud8.osic.org to point to endpoints and over https? that's what you're asking? | 21:45 |
| pabelanger | fungi: ya, I have that fixed locally | 21:45 |
| pabelanger | inc0: right, we are just talking if we should be using https | 21:45 |
| pabelanger | and dns | 21:45 |
| fungi | pabelanger: heh, i thought most letsencrypt clients renewed around 1-2 months so they didn't get anywhere near the 3-month expiration unless something broke | 21:45 |
| clarkb | pabelanger: I think we should most definitely use https | 21:46 |
| *** amitgandhinz has quit IRC | 21:46 | |
| clarkb | and dns | 21:46 |
| clarkb | just makes life simpler when DC ops decides they need to reip you for random reasons | 21:46 |
| pabelanger | inc0: looks like yes to both dns and https :) | 21:46 |
| pabelanger | clarkb: ++ | 21:46 |
| clarkb | or ship your hardware across the country :) | 21:46 |
| pabelanger | yes | 21:47 |
| *** dtardivel has quit IRC | 21:47 | |
| pabelanger | okay, stepping away. Will pick up when I get back | 21:47 |
| openstackgerrit | Steve McLellan proposed openstack-infra/project-config: Mark searchlight ES-1 test job non-voting https://review.openstack.org/356757 | 21:48 |
| cloudnull | fungi: our cert is on an F5 and we cant install the client there so its a manual client call and cert update. | 21:48 |
| fungi | cloudnull: aha, got it | 21:48 |
| clarkb | someone was telling me you have to run the client as root because it has to listen on some privileged port? | 21:48 |
| cloudnull | yes | 21:48 |
| cloudnull | ... | 21:48 |
| clarkb | maybe I misunderstood, but I can see how something like that would cause problems with $appliance | 21:48 |
| fungi | you can get root and install/run things on the control plane of an f5 big-ip, but i'm not sure i would want to | 21:49 |
| cloudnull | yes, i could do that. | 21:50 |
| clarkb | fungi: you could, but I remember them moving away from that | 21:50 |
| cloudnull | but the RAX netsec people would likely want to kill me | 21:50 |
| clarkb | they made some new shell thing you are supposed to use that is suppsoed to prevent you from doing that | 21:50 |
| fungi | clarkb: heehee | 21:50 |
| fungi | er, that was at cloudnull | 21:50 |
| *** edtubill has quit IRC | 21:51 | |
| clarkb | fungi: pabelanger so for the git repos being unhappy do we just need to manually push into them? | 21:53 |
| fungi | clarkb: if it's just that one repo, then that's probably the sanest fix for now | 21:53 |
| *** adriant has joined #openstack-infra | 21:54 | |
| clarkb | and for the github retries for replication I am guessing we just exceeded the api limit there and rerunning jeepyb would address that | 21:54 |
| *** thorst_ has quit IRC | 21:54 | |
| clarkb | though its like 5k api calls per hour and with 1k projects we may be pretty close to needing to better tune that | 21:55 |
| amrith | fungi, clarkb mordred ... thx for the help earlier. my ipv6 issue was in my own network. | 21:55 |
| amrith | resolved now | 21:55 |
| amrith | thx | 21:55 |
| cloudnull | woot! IPv6 all the things! | 21:56 |
| *** Hal has joined #openstack-infra | 21:57 | |
| *** mriedem has quit IRC | 21:58 | |
| *** devkulkarni has quit IRC | 21:59 | |
| *** shashank_hegde has quit IRC | 22:01 | |
| *** mdrabe has quit IRC | 22:02 | |
| clarkb | fungi: I am working through the deb- list to see if any others look unhappy | 22:02 |
| *** nikeshm has quit IRC | 22:04 | |
| *** valderrv has quit IRC | 22:04 | |
| *** sjmc7 has joined #openstack-infra | 22:04 | |
| sjmc7 | afternoon. i’m a bit rusty - is https://review.openstack.org/#/c/356757 sufficient to make that job non-voting? it’s run as part of the python-jobs-searchlight group | 22:06 |
| *** sdake_ has quit IRC | 22:06 | |
| mrhillsman | looking | 22:07 |
| *** kzaitsev_mb has quit IRC | 22:08 | |
| *** jordanP has quit IRC | 22:08 | |
| *** amotoki has joined #openstack-infra | 22:09 | |
| sjmc7 | thanks. i seem to have a mental block with zuul config | 22:09 |
| mrhillsman | i can setup letsencrypt pretty sure | 22:09 |
| *** gyee_ has quit IRC | 22:09 | |
| clarkb | sjmc7: yes that should do it, it would be great if we could refactor that job definition in jjb so that its a little bit less different than the other python27 jobs | 22:10 |
| clarkb | sjmc7: maybe use bindep or something to get es pulled in | 22:10 |
| *** amotoki_ has joined #openstack-infra | 22:10 | |
| *** shashank_hegde has joined #openstack-infra | 22:11 | |
| sjmc7 | yeah, i agree - i am going to look at that. we’re trying to test with two different versions of elasticsearch and as of today the 1.x test are failing, so we may end up dropping it entirely and installing 2.x through bindeps | 22:11 |
| sjmc7 | right now i need to get the gate unblocked | 22:11 |
| *** kzaitsev_mb has joined #openstack-infra | 22:12 | |
| clarkb | fungi: yup only deb-websockify seems unhappy | 22:12 |
| clarkb | fungi: I will work on pushing to it now | 22:12 |
| *** esberglu has quit IRC | 22:14 | |
| *** inc0 has quit IRC | 22:14 | |
| *** amotoki has quit IRC | 22:14 | |
| *** ayoung has quit IRC | 22:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Nodepool elements: Add a script to initialize urandom https://review.openstack.org/356697 | 22:16 |
| jeblair | clarkb, fungi, cloudnull, mordred: ^ updated | 22:16 |
| clarkb | getting fatal: Unpack error, check server log locally | 22:17 |
| *** spzala has joined #openstack-infra | 22:17 | |
| *** spzala has joined #openstack-infra | 22:17 | |
| cloudnull | jeblair: sorry. i had a late comment there on PS 1 | 22:17 |
| cloudnull | https://review.openstack.org/#/c/356697/1/nodepool/elements/initialize-urandom/static/usr/local/bin/initialize-urandom.py | 22:18 |
| cloudnull | seems trusty and centos7 return -1 while xenial returns 1 | 22:18 |
| *** elo has joined #openstack-infra | 22:18 | |
| jeblair | cloudnull: this only workes in kernel >= 3.17 (and < 4.something). but it's also only needed for that/those versions | 22:18 |
| jeblair | cloudnull: so we'll only be running this on xenial | 22:19 |
| clarkb | Caused by: org.eclipse.jgit.errors.CorruptObjectException: Invalid tree 7a80ca0841: mode starts with '0' | 22:19 |
| cloudnull | ah, makes sense. | 22:19 |
| clarkb | I think this is a case where old git allowed behavior that is invalid and new git explodes on it | 22:19 |
| clarkb | zigo: ^ you may need to repair this repo before we import it | 22:19 |
| clarkb | jeblair: thats beacuse getrandom is a new syscall ya? | 22:20 |
| jeblair | clarkb: yep, introduced in 3.17 | 22:20 |
| jeblair | gah, there's a comment that's wrong | 22:21 |
| clarkb | https://groups.google.com/forum/#!topic/repo-discuss/O_TaspFfY7s | 22:21 |
| *** sdake has joined #openstack-infra | 22:21 | |
| clarkb | git fsck --full shows warning in tree 7a80ca084167b04a732640a2f18ec86af55377fb: zeroPaddedFilemode: contains zero-padded file modes | 22:22 |
| clarkb | maybe we should add that to jeepyb | 22:22 |
| clarkb | er add a git fsck --full | 22:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Nodepool elements: Add a script to initialize urandom https://review.openstack.org/356697 | 22:22 |
| *** hockeynut has quit IRC | 22:22 | |
| clarkb | the google groups thread says we need to push directly into the on disk repo | 22:23 |
| clarkb | which I am worried will cause problems with indexing and may have other side effects we won't like | 22:23 |
| *** vgridnev has quit IRC | 22:23 | |
| *** timello has quit IRC | 22:23 | |
| *** lucas-dinner has quit IRC | 22:23 | |
| mordred | clarkb: yah - that's a thing that I'd want to test before doing it | 22:23 |
| openstackgerrit | Merged openstack-infra/project-config: Run ovo multinode grenade on 2 node label https://review.openstack.org/356592 | 22:23 |
| jeblair | i would prefer we not do it | 22:24 |
| fungi | clarkb: could we push an initialized but empty repo in and then push the fixed repo over top of it once available, so that we don't continue breaking image builds in the interim? | 22:24 |
| clarkb | ya I agree I don't want to do that and deal with more mysterious problems later | 22:25 |
| *** wcriswell has quit IRC | 22:25 | |
| *** eeiden has quit IRC | 22:25 | |
| jeblair | (i would prefer that we not import this repo in its current state, and that we do add the check to jeepyb) | 22:25 |
| clarkb | fungi: maybe | 22:25 |
| fungi | or maybe we can remove it from projects.yaml temporarily | 22:25 |
| *** jpeeler has quit IRC | 22:25 | |
| clarkb | fungi: basically push a master that is a dummy commit so gerrit and things are happy | 22:25 |
| *** ansiwen has quit IRC | 22:25 | |
| *** mdbooth has quit IRC | 22:25 | |
| fungi | that's probably sufficient to make image builds happy | 22:25 |
| *** burgerk has quit IRC | 22:25 | |
| *** stewie925 has joined #openstack-infra | 22:26 | |
| *** doffm has quit IRC | 22:26 | |
| *** doffm has joined #openstack-infra | 22:27 | |
| *** mdbooth has joined #openstack-infra | 22:27 | |
| *** ansiwen has joined #openstack-infra | 22:27 | |
| clarkb | https://www.perforce.com/blog/150109/git-out-vulnerability-out-fire explains how to fix | 22:27 |
| clarkb | I could run that fix (which will change all the sha1s) and push the result | 22:28 |
| clarkb | zigo: ^ you around? | 22:28 |
| jhesketh | Morning | 22:28 |
| clarkb | or maybe zigo can fix it on his end using that "trick" and then check it is good for him then we can grab it | 22:28 |
| *** pilgrimstack has joined #openstack-infra | 22:29 | |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: cover more combinations of version https://review.openstack.org/347045 | 22:30 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: remove stale lithium task https://review.openstack.org/356790 | 22:30 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: add experimental grenade job https://review.openstack.org/356791 | 22:30 |
| *** lucasagomes has joined #openstack-infra | 22:30 | |
| *** tphummel has quit IRC | 22:30 | |
| *** tphummel has joined #openstack-infra | 22:31 | |
| clarkb | I like that idea most I think | 22:31 |
| clarkb | that way we don't push a bunch of stuff zigo isn't expecting | 22:31 |
| fungi | in the interim, we still need to be able to build nodepool images, so probably need to remove the project from projects.yaml | 22:32 |
| *** acoles is now known as acoles_ | 22:33 | |
| clarkb | ya let me double check we generate the list from projects.yaml and not from gerrit ls-projects or similar | 22:33 |
| *** pilgrimstack has quit IRC | 22:33 | |
| clarkb | yup uses projects.yaml https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list#n32 let me push the change to remove deb-websockify for now | 22:34 |
| *** tphummel has quit IRC | 22:35 | |
| *** kzaitsev_mb has quit IRC | 22:35 | |
| *** gyee_ has joined #openstack-infra | 22:35 | |
| *** fguillot has joined #openstack-infra | 22:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Remove deb-websockify from projects.yaml https://review.openstack.org/356795 | 22:37 |
| clarkb | fungi: ^ | 22:37 |
| *** fguillot has quit IRC | 22:38 | |
| *** sdake has quit IRC | 22:38 | |
| *** jpeeler has joined #openstack-infra | 22:39 | |
| *** weshay has quit IRC | 22:41 | |
| clarkb | git fsck doesn't seem to have a treat warnings as errors option so will need to parse the output in jeepyb | 22:41 |
| *** beagles_biab is now known as beagles | 22:42 | |
| *** rbrndt has quit IRC | 22:45 | |
| *** xarses_ has quit IRC | 22:45 | |
| *** rbrndt has joined #openstack-infra | 22:46 | |
| *** rbrndt has quit IRC | 22:47 | |
| *** nwkarste_ has quit IRC | 22:49 | |
| *** eeiden has joined #openstack-infra | 22:49 | |
| *** nwkarsten has joined #openstack-infra | 22:50 | |
| *** amotoki_ has quit IRC | 22:50 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 22:51 |
| *** vhosakot has quit IRC | 22:53 | |
| *** timello has joined #openstack-infra | 22:54 | |
| *** nwkarsten has quit IRC | 22:55 | |
| *** wcriswell has joined #openstack-infra | 22:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/jeepyb: Check git fsck output when importing projects https://review.openstack.org/356800 | 22:56 |
| clarkb | jeblair: ^ thats not tested but should get us fsck checking in jeepyb | 22:56 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add Tap-as-a-service to PROJECTS https://review.openstack.org/356801 | 22:57 |
| clarkb | now to rereview entropy change | 22:58 |
| *** spzala has quit IRC | 22:58 | |
| *** spzala has joined #openstack-infra | 22:58 | |
| *** hongbin has quit IRC | 22:58 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Try increasing KeystoneWorkers to improve performance https://review.openstack.org/354606 | 23:00 |
| *** spzala has quit IRC | 23:03 | |
| clarkb | jeblair: looks like we are just going to let the non eagain errors bubble all the way out and kill the process ya? that seems like a sane way to address cloudnull's comment | 23:03 |
| jeblair | clarkb: yep | 23:03 |
| fungi | sometimes the most efficient way to fail is catastrophically | 23:04 |
| cloudnull | +1 | 23:04 |
| pabelanger | #status log osic-cloud8 credentials added to hieradata | 23:04 |
| openstackstatus | pabelanger: finished logging | 23:04 |
| clarkb | https://review.openstack.org/#/c/356706/1 is still missing the thing to cause it to run on boot though. Do we want to go ahead and approve it now or add that piece in? | 23:04 |
| *** asettle has joined #openstack-infra | 23:05 | |
| pabelanger | I haven't tested the systemd script yet, I can do that now | 23:05 |
| *** kzaitsev_mb has joined #openstack-infra | 23:06 | |
| jeblair | pabelanger: i'm done with that host again if you want it back :) | 23:06 |
| pabelanger | jeblair: sure, did you install your python app some place? | 23:06 |
| jeblair | pabelanger: ~root/random.py :) | 23:06 |
| pabelanger | thanks | 23:06 |
| jeblair | pabelanger: the existing element will put it in /usr/local/bin/initialize-urandom.py so you might want to start by copying it there | 23:07 |
| pabelanger | sure | 23:07 |
| jeblair | pabelanger: also, note that i think i left haveged disabled at boot, so you might want to 'update-rc.d enable' it | 23:07 |
| *** sshnaidm is now known as sshnaidm|afk | 23:08 | |
| *** amotoki has joined #openstack-infra | 23:08 | |
| cloudnull | based on https://review.openstack.org/#/c/356706/1/nodepool/nodepool.yaml do we want the to run the initialize-urandom on all of the images? wouldn't that execute "/usr/local/bin/initialize-urandom.py" everywhere? | 23:08 |
| cloudnull | ^ sorry, may need more coffee | 23:08 |
| jeblair | cloudnull: good point, it should just be ubuntu-xenial | 23:09 |
| *** asettle has quit IRC | 23:09 | |
| fungi | ahh, yep, i didn't think about the fact that we can just _not_ install it except on the xenial images | 23:10 |
| clarkb | ya I left the install script change alone | 23:10 |
| fungi | simply by not including that element | 23:10 |
| *** xyang1 has quit IRC | 23:10 | |
| cloudnull | maybe it should run on fedora and xenial ? | 23:10 |
| clarkb | only approved the script add change | 23:10 |
| jeblair | it will harmlessly catastrophically fail elsewhere :) | 23:10 |
| cloudnull | I gotta run, talk to you all later, may be back online in a while | 23:11 |
| *** amotoki has quit IRC | 23:13 | |
| *** kzaitsev_mb has quit IRC | 23:15 | |
| *** phschwartz has joined #openstack-infra | 23:15 | |
| fungi | Krenair: jpmaxman: today ended up not being conducive for trying to stand up a replacement wiki server with the proposed puppet changes, so i'm afraid i won't get to it until tomorrow at this point | 23:15 |
| fungi | have guests coming into town and need to go be social now | 23:15 |
| fungi | back later | 23:15 |
| jpmaxman | fungi no worries | 23:16 |
| Krenair | ok | 23:16 |
| jpmaxman | if there is anything I can do to help let me know | 23:16 |
| pabelanger | okay, systemd script worked | 23:16 |
| pabelanger | just making sure I have the targets setup properly | 23:17 |
| *** tpsilva has quit IRC | 23:18 | |
| *** ayoung has joined #openstack-infra | 23:18 | |
| *** fguillot has joined #openstack-infra | 23:20 | |
| *** gildub has joined #openstack-infra | 23:21 | |
| *** dingyichen has joined #openstack-infra | 23:27 | |
| *** kbaegis1 has quit IRC | 23:27 | |
| *** gouthamr has joined #openstack-infra | 23:28 | |
| *** shashank_hegde has quit IRC | 23:29 | |
| *** pilgrimstack has joined #openstack-infra | 23:30 | |
| *** sjmc7 has quit IRC | 23:31 | |
| *** rbuzatu has joined #openstack-infra | 23:32 | |
| *** baoli has quit IRC | 23:33 | |
| *** pilgrimstack has quit IRC | 23:35 | |
| pabelanger | jeblair: http://paste.openstack.org/show/560281/ Does that look right? | 23:35 |
| *** amotoki has joined #openstack-infra | 23:35 | |
| *** rbuzatu has quit IRC | 23:37 | |
| *** annegentle has quit IRC | 23:40 | |
| *** amotoki has quit IRC | 23:41 | |
| *** klindgren has joined #openstack-infra | 23:41 | |
| *** Swami has quit IRC | 23:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add glue to get initialize-urandom installed https://review.openstack.org/356706 | 23:42 |
| pabelanger | mordred: jeblair: clarkb: fungi: ^ adds systemd bits | 23:43 |
| *** nwkarsten has joined #openstack-infra | 23:43 | |
| pabelanger | I haven't build DIB yet, but I used growroot element as reference | 23:43 |
| pabelanger | as you can tell by my typo | 23:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add glue to get initialize-urandom installed https://review.openstack.org/356706 | 23:44 |
| *** hichihara has joined #openstack-infra | 23:46 | |
| *** amitgandhinz has joined #openstack-infra | 23:47 | |
| *** nwkarsten has quit IRC | 23:50 | |
| *** zhurong has joined #openstack-infra | 23:51 | |
| *** rwsu has quit IRC | 23:51 | |
| *** aeng has quit IRC | 23:51 | |
| jeblair | pabelanger: yes, that output looks good | 23:51 |
| *** valderrv has joined #openstack-infra | 23:51 | |
| pabelanger | jeblair: great | 23:52 |
| *** amitgandhinz has quit IRC | 23:52 | |
| jeblair | pabelanger: is it possible to run both before haveged and unbound? | 23:53 |
| jeblair | pabelanger: mostly i would like to avoid having to think about what it means to have two haveged processes sending data to the kernel at the same time (maybe it's okay? i don't know) | 23:54 |
| *** shashank_hegde has joined #openstack-infra | 23:54 | |
| jeblair | that question would have been better englished as "is it possible for initialize-urandom to run before both havegad and unbound?" | 23:55 |
| pabelanger | jeblair: Oh, ya. I thought havegad was a dependency | 23:55 |
| jeblair | pabelanger: only that it needs to be installed, not running (the python program starts its own process to get the data and stops it on exit) | 23:56 |
| pabelanger | we can do before=network-pre.target | 23:56 |
| pabelanger | and think we'll be fine | 23:56 |
| pabelanger | okay, let me test and update | 23:56 |
| openstackgerrit | Merged openstack-infra/project-config: Nodepool elements: Add a script to initialize urandom https://review.openstack.org/356697 | 23:57 |
| *** markusry has quit IRC | 23:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove deb-websockify from projects.yaml https://review.openstack.org/356795 | 23:57 |
| *** kzaitsev_mb has joined #openstack-infra | 23:57 | |
| *** hpe-hj has quit IRC | 23:58 | |
| clarkb | jeblair: I would expect that the kernel would handle that for us? but I guess there may be some concern about not creating proper entropy in that case | 23:58 |
| clarkb | repeating bits or similar | 23:58 |
| jeblair | clarkb: oh yeah the kernel is fine -- my question is more: will 2 havegeds output the same or similar bits? | 23:59 |
| clarkb | ya | 23:59 |
| jeblair | i haven't read the algorithm at all, so i literally have no idea :) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!