Wednesday, 2014-03-19

greghaynesyep, looks like they call it i68600:00
*** CaptTofu has quit IRC00:00
*** CaptTofu has joined #tripleo00:00
clarkbare you guys still using i386 for memory constraints?00:00
Lotus907eficlarkb: before yu completely dismiss the httpretty bug see the note at the bottom of the bug report please00:00
clarkbLotus907efi: which note?00:01
* greghaynes wonders if lifeless will not experience the OOM on x86 also00:01
Lotus907efionce httpretty fails to build the command that executes afterwards takes f'in FOREVER to complete00:01
clarkbLotus907efi: right it is continuing on and do all of the othe rwheels00:01
clarkbLotus907efi: at least that is what I would expect it to do. which means compiling stuff00:01
Lotus907efiyes, but should the next command take 10 - 15 minutes for ONE thing?00:01
clarkbLotus907efi: you can check by seeing if you get any wheels from packages listed after httpretty00:01
clarkbLotus907efi: sure its literally compiling all of the requirements so that it can build them into wheels00:02
jogolifeless: what version of libvirt do you guys use again?00:02
Lotus907efion a 24 core machine with 24GB of memory?00:02
jogolifeless: this is about which version of libvirt to test in devstack-gate00:02
clarkbLotus907efi: sure, it doesn't necessarily do the compiles in parallel00:03
Lotus907efihmmm, ok00:03
*** andreaf has joined #tripleo00:03
Lotus907eficlarkb: I just checked two packages that occur AFTER httpretty00:04
Lotus907efino whl files00:04
Lotus907efilet me continue checking00:05
Lotus907efihmm, out of 6 - 8 things after httpretty only one I have checked has a whl file00:06
Lotus907efiboto00:06
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Use xtradb-cluster in place of mysql  https://review.openstack.org/7554000:07
clarkbLotus907efi: ok I spent a bit more time reading the code. pypi-mirror seems to do a best effort with the wheel command00:07
clarkbLotus907efi: it then does the tarballs and only that return code is checked. So it should continue on just fine00:07
clarkbbut not everything may get a wheel00:07
Lotus907efihmmm, ok00:08
Lotus907efiso what was the answer to my question about pip having a whl file but no tar.gz file?00:09
clarkbLotus907efi: https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n294 what happens there is it says give me a list of everyting I did manage to install now let us get the tarball and wheels for those things00:09
clarkbthe wheel step may fail again00:09
Lotus907efiI know you said pip and some other things were treated as special cases00:09
Lotus907efidue to dependencies00:09
clarkbLotus907efi: yes see scrollback above where I like to the code00:09
Lotus907efithe bit about upgrade_format and the no-use-wheel thing?00:11
clarkbyes00:11
Lotus907efiso for special cases like pip it would NOT use a whl file and make sure the tar.gz file was pulled down00:12
lifelessjogo: saucies00:13
lifelessjogo: today00:13
lifelessgreghaynes: re'll find out :)00:14
clarkbLotus907efi: current code appears to just use a wheel00:14
clarkbLotus907efi: so I think the httpretty bug is invalid but the pip bug is valid00:14
lifelessclarkb: our CI servers do amd6400:14
Lotus907efihmm, ok00:14
Lotus907efisounds right00:14
greghayneslifeless: I just pushed what *should* fix it00:14
greghaynesgotta wait till my devtest with another patch finishes00:14
jogolifeless: thanks that would be 1.1.100:15
Lotus907efigreghaynes: how long do your devtest.sh runs take on average? and are you using internal mirrors?00:15
greghaynesLotus907efi: mine are about 30ish mins00:15
greghaynesand yes, mirrors00:15
Lotus907efiwow, that is pretty fast00:15
greghaynesraid0 ssd with noatime for great good00:16
Lotus907efican I speed this up even more if we parallelize the code on a HPC cluster?00:16
Lotus907efiI might have 1500 nodes hanging around in a few weeks :)00:16
lifelessgreghaynes: thats happier00:17
greghaynes*shrug* - I did hear about someone tmpfs'ing to speed it up00:17
Lotus907efiyeah, that would help too00:17
Lotus907efiwe do have some BIG nodes here in hpcloud testing rack with 148GB of memory00:17
lifelessgreghaynes: relatime should be fine00:17
greghaynesprobably want more than 24g ram for that00:17
greghaynesYea, I noatime so my ssd's (in theory) live longer00:17
Lotus907efiI think they are DL380 G8 systems00:18
Lotus907efiwhen I was working at Intel a few years ago they had some HPC nodes that had over 300GB of memory00:19
greghaynesheh00:19
tchaypoand then for a really fast build, make sure you put your swap file on tmpfs too00:19
Lotus907efihehehehe00:19
greghaynesmy deal is just 32gb ram, but might be able to tmpfs that too00:19
StevenKtchaypo: Or just ignore swap00:20
* greghaynes facepalms @ tchaypo00:20
greghaynesyes, like why even have swap at that point :p00:20
Lotus907efiyeah, on those systems there was no swap space00:20
*** matsuhashi has joined #tripleo00:20
tchaypothank you greghaynes :)00:20
greghayneshehe00:20
tchaypoStevenK: ah, you're awake!00:20
Lotus907efino, this version of StevenK is just a dream within a dream00:21
StevenKtchaypo: Have been for a few hours?00:21
lifelessgreghaynes: you have swap space?00:23
* greghaynes looks00:23
greghaynesyes :(00:23
StevenKI also have swap space on my laptop00:23
StevenKWhich I think I might kill00:24
greghaynesWell, theres really no excuse - this is box dedicated to basically running devtest :p00:24
Lotus907efiI think my swap space is pointed at /dev/null00:24
Lotus907efimy system really speeded up when I did that00:24
StevenKLotus907efi: Hm, swapping pages back into RAM must be fun, then.00:25
greghaynesswap as a replacement for memset(0, ...)00:25
Lotus907efiin the 1930's when an upitty customer complained to Ettore Bugatti about the brakes on one of his cars, he replied that brakes just slow you down00:26
clarkbLotus907efi: ok pypi-mirror bugs updated. Feel free to take a shot at fixing the one for pip and wheels00:27
Lotus907efiwhen another customer complained about his car being hard to start when cold Bugatti said if he couldn't afford a heated garage then he shouldn't buy a Bugatti00:27
*** CaptTofu has quit IRC00:27
lifelessgreghaynes: so00:28
lifeless 3406 mysql     20   0  827m 104m 6820 S   0.3  5.2   0:01.21 mysqld00:28
lifelessgreghaynes: is that what you're seeing?00:28
lifelessgreghaynes: when does the memory bloat kick in and break the world ?00:28
greghaynesNo. For me it kicked in while it setup the undercloud (os-c-c run)... I wonder if your running 32bit makes a big difference though00:29
lifelessok, odd though as o-c-c runs on seed too00:30
lifelessbut specifically one doesn't need to build an overcloud to reproduce?00:30
greghaynesyep, seed didnt oom for me though for who knows why00:30
* greghaynes kicks of devtest run with xtradb patch00:30
*** rpodolyaka1 has joined #tripleo00:40
*** andreaf has quit IRC00:41
Lotus907efiso if you switch to xtradb won't that sort of invalidate all of the work and patches Thersa MChale has been submitting against mysql ?00:42
greghaynesPossibly some00:42
greghaynesalthough I sent the xtradb patch to her and shes been following it00:43
lifelessoh great00:43
lifeless*now* I trigger the infamous ext4 resize bug00:43
StevenKWhich bug?00:43
lifelessI thought we merged dan's fix for it00:44
greghaynesI didnt realize that was still around00:44
*** rpodolyaka1 has quit IRC00:44
lifelessgreghaynes: http://imgur.com/Z47nEKZ00:46
greghaynesyerp00:46
*** yamahata has quit IRC00:47
* lifeless respins the image00:50
*** nosnos has joined #tripleo00:51
*** nosnos_ has joined #tripleo00:53
*** nosnos has quit IRC00:57
greghayneshrm, for some reason my $TE_DATAFILE is not valid json...01:01
greghaynesspecifically, undersloud ssh-key has actual newlines in it01:02
StevenKlifeless: Can you +A https://review.openstack.org/#/c/75444/ ? It's had 2 +2's for > a week01:04
Lotus907efihmm, I cannot get an undersloud to build, all I end up with are all these damn underclouds01:06
StevenKgreghaynes: Hm. I have a patch up that uses $(cat) inside jq, and that has literal '\n's in the string01:06
Lotus907efijq? I heard that mentioned somewhere else today01:06
Lotus907efiwhat is jq01:07
greghaynesman jq :p01:07
Lotus907efiis that a debian package perhaps?01:07
greghaynesits... interesting01:07
Lotus907efiah01:07
Lotus907efiok01:07
StevenKLotus907efi: http://stedolan.github.io/jq/01:07
Lotus907efijson preprocessor01:07
Lotus907efiso jq is a debian unstable package01:08
Lotus907efiand if some un-named "company" only allows pulls from Debian testing then this package is unavailable01:08
lifelessyes01:08
lifelessusing testing is generally a terrible idea01:08
lifelessuse unstable or stable only01:08
lifelesstesting has nearly all the flaws of unstable but not security updates01:09
lifelesslast I heard01:09
Lotus907efiwell a certain new project at some here-to-for un-named company has decided to pull from debian testing01:09
clarkblifeless: I heard a rumor you may have helped make a distro thing to fix that01:09
lifelessclarkb: Ubuntu ?01:10
clarkbya :)01:10
lifelessfixish :P01:10
Lotus907efifor it's new orthogonal distro01:10
lifelessLotus907efi: are you encountering an error about a missing jq ?01:11
Lotus907efiand since jq is not in debian testing then it is not available to test tripleo with this new orthogonal distro01:11
Lotus907efiI will as soon as I start testing with "orthogonal"01:11
StevenKIt failed to build on armhf01:11
lifelessLotus907efi: you should be able to just use debian unstable urls01:11
Lotus907efiok, well I guess that is possible01:12
Lotus907efianyway, goodnight all, I have had enough fun for one day01:16
*** Lotus907efi is now known as Lotus907efi_01:17
openstackgerritA change was merged to openstack/diskimage-builder: Update openSUSE package mapping for libffi-dev  https://review.openstack.org/7544401:18
*** tserong has quit IRC01:33
*** rpodolyaka1 has joined #tripleo01:40
lifelessoh01:41
lifelessI've got a new theory on this resize2fs bug01:41
lifelesswhat if the lazy metadata hasn't finished lazy updating before we unmount and freeze the image01:41
lifelessdoes resize2fs handle that...01:42
*** tserong has joined #tripleo01:46
*** tserong has joined #tripleo01:46
*** rpodolyaka1 has quit IRC01:46
lifelessgreghaynes: undercloud looks fine01:50
lifeless 3771 mysql     20   0  828m 104m 6772 S   0.0  5.2   0:02.62 mysqld01:50
lifelessI'll try AMD6401:50
openstackgerritGregory Haynes proposed a change to openstack/tripleo-image-elements: Explicitly pip install pbr before its installed as dependency  https://review.openstack.org/8110101:54
greghayneshrm, and CI just had a triple SUCCESS on the xtradb patch01:55
*** dkehn has quit IRC01:58
*** dkehn has joined #tripleo02:07
*** yamahata has joined #tripleo02:07
openstackgerritA change was merged to openstack/tripleo-incubator: Use the DIB_REPOLOCATION / DIB_REPOREF for the bootstrap.  https://review.openstack.org/7837102:18
openstackgerritA change was merged to openstack/tripleo-incubator: Switch to using jq to write out initial JSON  https://review.openstack.org/8088402:20
*** CaptTofu has joined #tripleo02:25
*** shakamunyi has joined #tripleo02:28
*** fandi has joined #tripleo02:33
StevenKlifeless: Do you still have reservations about https://review.openstack.org/#/c/77741/ ?02:40
lifelessI never had reservations02:42
lifelessI just wanted to know it worked ;)02:42
openstackgerritA change was merged to openstack-infra/tripleo-ci: Switch to nodes list in the JSON  https://review.openstack.org/7774102:42
lifelessStevenK: whats up with https://review.openstack.org/#/c/79051/ ?02:43
*** cwolferh has quit IRC02:44
StevenKlifeless: Let me rebase that02:47
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Store seed details in the JSON  https://review.openstack.org/7905102:48
StevenKlifeless: I was plotting to finish off the remove backward compatability branch02:50
lifelessStevenK: cool. we'll need to deploy a new set of testenvs02:52
StevenKlifeless: Sure, that branch won't pass CI until we do, so that can happen whenever02:54
lifelessokydoky02:55
StevenKSigh02:57
StevenKTalking to saved VMs isn't very helpful02:57
* StevenK wants the RAM he ordered already02:59
StevenKHah, I think I can see why I can't talk to the VMs, even though they're running03:01
lifelessENOROUTE03:01
StevenKExactly03:01
StevenKTrying to work out how to resurrect that bit03:01
lifelessip route add 192.0.2.0/24 via $SEEDIP03:02
*** matsuhashi has quit IRC03:13
lifelessbuilding new te image now03:17
*** matsuhashi has joined #tripleo03:20
*** matsuhashi has quit IRC03:25
*** lazy_prince has quit IRC03:28
*** xuhaiwei has quit IRC03:30
*** killer_prince has joined #tripleo03:34
lifelessok we have new te imagefs we can deploy03:35
lifelessjust need to take down CI to do it ...03:35
*** weshay has quit IRC03:39
*** nosnos_ has quit IRC03:41
*** rpodolyaka1 has joined #tripleo03:44
*** rpodolyaka1 has quit IRC03:48
*** killer_prince has quit IRC03:54
*** vkozhukalov_ has joined #tripleo03:55
openstackgerritRichard Su proposed a change to openstack/tripleo-image-elements: Update selinux policies for mysqld  https://review.openstack.org/7460703:55
*** ramishra has joined #tripleo04:00
*** CaptTofu has quit IRC04:02
*** CaptTofu has joined #tripleo04:03
*** CaptTofu has quit IRC04:07
*** xuhaiwei has joined #tripleo04:10
*** matsuhashi has joined #tripleo04:23
*** nosnos has joined #tripleo04:35
*** rpodolyaka1 has joined #tripleo04:45
openstackgerritSteve Kowalik proposed a change to openstack-infra/tripleo-ci: Change how the JSON is updated for readability  https://review.openstack.org/8090504:46
*** rpodolyaka1 has quit IRC04:49
openstackgerritSteve Kowalik proposed a change to openstack-infra/tripleo-ci: Populate seed.ip in the testenv JSON  https://review.openstack.org/7904304:51
tchaypoStevenK: https://bugs.launchpad.net/tripleo/+bug/129408304:55
StevenKYeah, which you saw yesterday04:57
*** gcha has quit IRC04:57
tchaypowhich causes me to take a look at devtest.sh to see how --existing-environment works04:59
tchaypoand the answer is that it skips running _testenv entirely04:59
tchaypoand the other fun I've just found is from05:01
tchaypo13:18 < openstackgerrit> A change was merged to openstack/tripleo-incubator: Use the DIB_REPOLOCATION / DIB_REPOREF for the bootstrap.  https://review.openstack.org/7837105:01
tchaypoit turns out that change breaks if DIB_REPOLOCATION_diskimage_builder is not set05:02
StevenKYay?05:03
lifelessdamn05:05
lifelessgoing to revert or fix?05:05
tchaypoI'm going to spend about 15 minutes trying to fix, then revert05:05
tchaypoI'd prefer to fix but the lines I'm going to need to edit are already approaching line-noise05:06
tchaypohow do i test to see if a variable is set without tripping set -u?05:12
tchaypoI'm sure I've done this before. recently, even.05:12
lifelessyou can use ${foo:-''}05:15
lifelessor the other ${NAME: operators05:15
tchaypoyep, just like the source-repositories element does05:16
tchaypoactually it's even fancier; it uses ${!REPOLOCATION_OVERRIDE:-...}05:19
*** jesusaurus has quit IRC05:26
*** andreaf has joined #tripleo05:35
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref  https://review.openstack.org/8144305:37
tchaypolifeless, StevenK - ^^05:37
*** xuhaiwei has quit IRC05:37
*** Killer_prince has joined #tripleo05:37
* StevenK dabs at the blood coming from his eyes05:38
*** tchaypo changes topic to "FIREDRILL https://review.openstack.org/81443 | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"05:38
tchaypoDoes it count as a firedrill if it only breaks users and doesn't upset CI?05:38
tchaypooh blah05:41
tchaypodid you spot the glaring mistake?05:41
StevenKHeh05:43
StevenKYes05:43
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref  https://review.openstack.org/8144305:45
tchaypoof course, it would perfectly fine when I tested it, because I don't have any of those things set so it was using the defaults.05:45
*** rpodolyaka1 has joined #tripleo05:45
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref  https://review.openstack.org/8144305:46
*** andreaf has quit IRC05:48
*** rpodolyaka1 has quit IRC05:50
tchaypojang: Could you review that change when you get a chance? I'd like to confirm that it still does what you intended05:50
tchaypoit'd be really handy if I could just set DIB_REPOREF_tripleo_incubator and have pull-tools pull down my patch05:51
*** xuhaiwei has joined #tripleo05:53
tchaypono wait, that should actually work05:53
tchaypobecause that will get read by devtest_setup.sh, which works fine; and by the time it runs pull-tools it will have my version05:54
tchaypoah, nope. it won't re-pull the cached -incubator if it's already there05:59
*** matsuhashi has quit IRC05:59
StevenKIt should update it, though, no?05:59
tchaypoyes05:59
tchaypobut the update happens *after* the reference to the unbound DIB_REPOLOCATION_diskimage_builder05:59
lifelesstheres an interaction with ZUUL_REF too though05:59
tchaypoeven if -incubator were the first tool in $TOOLS, by that point you're already inside the pull-tools script and it's going to die when it hits the second one06:00
tchayponever mind. git review -d 81443 it is.06:00
*** matsuhashi has joined #tripleo06:01
*** rpodolyaka1 has joined #tripleo06:01
tchaypohttp://paste.openstack.org/show/1fe6CsjEsK6K3DvwCv4t/06:02
tchaypothat;s not really how I'd like devtest.sh --trash-my-machine to end06:03
*** nati_ueno has joined #tripleo06:03
*** ramishra has quit IRC06:03
*** CaptTofu has joined #tripleo06:03
tchaypoit runs06:06
tchaypo+ git --no-pager log -1 --pretty=oneline06:06
tchaypoand then exits06:06
tchaypoexit code is 0.. why does that abort?06:06
*** CaptTofu has quit IRC06:08
*** akuznetsov has quit IRC06:08
*** xuhaiwei has quit IRC06:11
*** ramishra has joined #tripleo06:12
lifelessmaster doesn't look like a branch, or06:14
lifelesstchaypo: bad checking on whether to pull ;)06:14
tchaypoyeah, that's what I'm poking at - line 4706:15
tchaypoI understand why it skips the pull, but why does that abort the whole devtest.sh execution?06:18
lifelessdoes it? the first one doesn't06:20
*** giulivo has quit IRC06:21
tchaypooh, I see what you mean, I think06:22
tchaypoI'm on review/james_polley/use_default_values_for_repo_locatoin_and_ref06:22
tchaypobut the error message says "master doesn't look like a branch"06:23
tchaypoie, it's using its default $REF even though I've manually moved it to a different branch.06:23
lifelessbbiaw06:25
*** akuznetsov has joined #tripleo06:27
*** nati_uen_ has joined #tripleo06:28
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref  https://review.openstack.org/8144306:29
*** nati_ueno has quit IRC06:31
*** jtomasek has joined #tripleo06:33
tchaypo*facepalm*06:34
tchaypothe reason it exists after that point is because I was running devtest_setup.sh not devtest.sh06:34
tchaypoand devtest_setup.sh doesn't have any more to do06:34
* tchaypo is so clever sometimes06:35
tchaypoand that's handy, because i can now run away for dinner with a clear conscience knowing that I have a patch that works.06:36
*** nati_uen_ has quit IRC06:38
*** Slower has quit IRC06:40
*** rlandy has joined #tripleo06:43
SpamapShm06:45
SpamapSso has anyone else tried to use a local mirror w/ the pypi mirror lately?06:45
SpamapSI got the PrettyTable fail06:45
SpamapSApparently pip expects that the index url will serve things in a case-insensitive manner..06:46
SpamapSbut our index url is file:/// ...06:46
SpamapSso.. not so much06:46
Killer_princeSpamapS: I did try using pypi local mirror.. seems to be working with dib. but doesnt seem to work with dib when used with devtest..06:47
SpamapSright06:47
SpamapShowever, serving it via apache with CheckSpelling on does work06:47
*** matsuhashi has quit IRC06:47
Killer_princehmm..06:47
SpamapSwhich makes sense..06:48
SpamapSbut what I really want is for pip to handle the fact that file:/// is case sensitive..06:49
SpamapSI think this is only showing up because we're using latest pip/virtualenv06:49
SpamapSmordred: ^^06:49
SpamapSclarkb: ^^06:49
*** matsuhashi has joined #tripleo06:50
*** ramishra has quit IRC06:52
*** Slower has joined #tripleo06:53
*** morazi has quit IRC06:56
SpamapShm06:57
SpamapSbuilding i386 images, but my local pypi mirror is x86-6406:57
SpamapSseems wheels is a bit scary to actually use.. :-P06:57
*** sseago has quit IRC06:58
lifelessSpamapS: case sensitive should be fine, my mirror is http which is case sensitive07:00
lifelessSpamapS: wheels are quite early in learning :)07:02
SpamapSlifeless: if I turn off CheckSpelling, I get a fail 100% of the time07:03
SpamapSlifeless: or if I use the default file:///home/clint/.cache/image-create/pypi/07:03
StevenKIs it still mod_speling07:04
StevenK?07:04
SpamapS127.0.0.1 - - [19/Mar/2014:00:00:22 -0700] "GET /pypi/mirror/Ubuntu-13.10/PrettyTable/ HTTP/1.1" 301 625 "-" "pip/1.5.4 CPython/2.7.5+ Linux/3.11.0-17-generic"07:04
SpamapS127.0.0.1 - - [19/Mar/2014:00:00:22 -0700] "GET /pypi/mirror/Ubuntu-13.10/prettytable/ HTTP/1.1" 200 507 "-" "pip/1.5.4 CPython/2.7.5+ Linux/3.11.0-17-generic"07:04
SpamapSStevenK: yes07:04
SpamapS(this behavior is the same whether I use the Ubuntu-13.10 wheels mirror or not)07:04
StevenKSpamapS: Oh, and the 301 is from mod_speling, and if it's off, the response is 404 instead?07:05
SpamapSI'm also seeing the same fail Lotus907efi_ was seeing.. easy_install can't seem to get pbr right07:05
SpamapS127.0.0.1 - - [19/Mar/2014:00:00:28 -0700] "GET /pypi/mirror/Ubuntu-13.10/pbr/ HTTP/1.1" 200 627 "-" "Python-urllib/2.7 setuptools/3.3"07:05
SpamapSStevenK: correct07:05
SpamapSStevenK: and there is no follow-up to prettytable07:05
StevenKloopback vfat filesystem? :-P07:05
SpamapSStevenK: don't think I haven't toyed with the idea. ;)07:06
StevenKHaha07:06
StevenKApparently, you can trick samba into doing it as well07:06
StevenKI can recall the topic of a blog post that I read a while back07:06
lifelessSpamapS: is it on by default ?07:07
StevenKmod_speling is not, IIRC07:07
SpamapSlifeless: it is not07:07
SpamapSyou have to a2enmod it07:07
SpamapSor load it elsewhere07:07
SpamapSlifeless: are you using --offline ?07:08
*** ramishra has joined #tripleo07:15
lifelessSpamapS: did you use pypi-mirror or bandersnatch to make your mirror ?07:15
lifelessSpamapS: because I'll be pypi-mirror07:15
lifelessand missing metadata07:15
lifelessor something07:16
lifelesshmm, nothing obvious to cause tat07:17
*** rpodolyaka1 has quit IRC07:17
SpamapSlifeless: pypi-mirror07:17
lifelessbut hitting e.g. prettytable in my mirror is 404 not magic or redirects07:17
SpamapSlifeless: PrettyTable is 404 for me07:17
SpamapSlifeless: so it may be an ordering thing?07:17
SpamapSas in..07:18
SpamapSglobal reqs may have changed from one to the other between our first runs07:18
lifelessSpamapS: do you mean, if we have PrettyTable somewhere07:19
lifelessonce its installed a bad ref to prettytable is ignored?07:19
lifelessbut this has worked for weeks for me07:19
SpamapSI made my pypi-mirror about a week ago07:19
SpamapSlifeless: on disk, do you have 'prettytable' or 'PrettyTable' ?07:20
SpamapSlifeless: I _only_ have 'prettytable'07:20
lifelessSpamapS: this is a bandersnatch mirror07:20
lifelessI only have PrettyTable07:20
SpamapSinteresting07:21
lifelesswhich since its bandersnatch07:22
lifelessmeans thats the official name07:22
SpamapSwhat if it is as simple as pypi.python.org/simple changed?07:22
lifelessand pypi-mirror is mirroring incorrectly for some reason07:22
SpamapSlifeless: there me be an invalid assumption..07:26
*** cwolferh has joined #tripleo07:27
SpamapSlifeless: https://pypi.python.org/pypi/PrettyTable <-- points at prettytable-* for download07:27
lifelessyes07:28
lifelessand07:28
SpamapSyep.. think I found it07:28
*** ramishra has quit IRC07:29
openstackgerritLadislav Smola proposed a change to openstack/tripleo-image-elements: Setting negative RamWeigher  https://review.openstack.org/8120007:29
*** jtomasek has quit IRC07:30
SpamapSlifeless: https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n37507:30
*** matsuhashi has quit IRC07:30
SpamapSShould be that package_name is used to create the index dir07:30
SpamapSerr07:30
SpamapSyeah07:30
*** matsuhashi has joined #tripleo07:31
lifelessisn't packagename being inferred there ?07:32
lifelessvia a regex thus the issue ?07:32
SpamapSso it is using the regex on the _file_ as the package name07:33
SpamapSand then creating a dir for the package name07:33
SpamapSinstead of just using the same parent dir07:33
SpamapSNot sure why pypi.openstack.org users don't see this07:35
SpamapSHEAD http://pypi.openstack.org/openstack/PrettyTable/07:35
SpamapSUser-Agent: lwp-request/6.03 libwww-perl/6.0507:35
SpamapS404 Not Found07:35
*** xuhaiwei has joined #tripleo07:37
xuhaiweilifeless: Can we use one undercloud to create more than one overcloud?07:40
*** morganfainberg is now known as morganfainberg_Z07:43
SpamapSxuhaiwei: yes definitely07:45
SpamapSxuhaiwei: we have done that for our CI purposes :)07:46
StevenKHuh, some of the CI jobs have failed with PrettyTable07:46
StevenKAnd some are passing07:46
xuhaiweican we do that(create more than one overcloud) using devtest?07:48
SpamapSxuhaiwei: no07:49
SpamapSxuhaiwei: but you can use the devtest created undercloud to manually drive Heat and create a second overcloud.07:49
SpamapSjang: https://review.openstack.org/81469 <-- the "spellcheck" problem solved.07:50
*** rdopieralski has joined #tripleo07:52
*** e0ne has joined #tripleo07:52
*** ramishra has joined #tripleo07:52
StevenKSpamapS: You can make that a little cleaner07:54
StevenKSpamapS: name_match is no longer referenced, so you can collapse it down to if PACKAGE_VERSION_RE.search(tarball) is None:07:54
*** nati_ueno has joined #tripleo07:55
*** ifarkas has quit IRC07:58
*** rdopieralski has quit IRC07:58
*** noslzzp has joined #tripleo08:02
*** CaptTofu has joined #tripleo08:04
*** athomas has joined #tripleo08:04
*** shakamunyi has quit IRC08:07
*** jtomasek has joined #tripleo08:08
GheRiveromorning TripleO!08:09
*** CaptTofu has quit IRC08:09
*** e0ne has quit IRC08:11
*** martyntaylor has joined #tripleo08:16
*** mrunge has joined #tripleo08:19
*** jcoufal has joined #tripleo08:22
rpodolyakamorning all08:22
*** rpodolyaka1 has joined #tripleo08:24
*** martyntaylor1 has joined #tripleo08:24
SpamapSStevenK: by all means, make it cleaner. I find the code a bit.. obtuse.. :-P08:25
*** uvirtbot has joined #tripleo08:25
*** martyntaylor has quit IRC08:26
SpamapSStevenK: however, it is time that I slept08:26
* SpamapS sleeps08:27
*** martyntaylor has joined #tripleo08:27
*** rpodolyaka1 has quit IRC08:28
*** martyntaylor1 has quit IRC08:30
*** shakamunyi has joined #tripleo08:33
*** sseago has joined #tripleo08:33
*** shakamunyi has quit IRC08:38
*** yamahata has quit IRC08:51
jangJust tried the latest pypi-mirror from git, by the way: it really does create tarballs alongside wheels, so that's rather an improvement. Thanks all.08:51
*** jcoufal has quit IRC08:54
*** andreaf has joined #tripleo08:54
*** vkozhukalov_ has quit IRC08:55
*** al has quit IRC08:55
*** al has joined #tripleo08:56
lifelessGheRivero: https://review.openstack.org/#/c/81443/08:57
lifelessGheRivero: (firedrill in topic)08:57
*** hashar has joined #tripleo08:57
GheRiverolifeless: looking08:57
*** bauzas has joined #tripleo09:02
*** e0ne has joined #tripleo09:03
GheRiverolifeless: is there a bug for the prettytable issue?09:07
openstackgerritA change was merged to openstack/tripleo-incubator: Set default values for repo location and ref  https://review.openstack.org/8144309:08
*** e0ne has quit IRC09:20
*** e0ne has joined #tripleo09:22
*** rpodolyaka1 has joined #tripleo09:24
*** jcoufal has joined #tripleo09:24
*** jcoufal has quit IRC09:25
*** jcoufal has joined #tripleo09:26
*** rpodolyaka1 has quit IRC09:28
*** jrist has joined #tripleo09:32
*** nati_ueno has quit IRC09:33
*** nati_ueno has joined #tripleo09:34
*** shakamunyi has joined #tripleo09:34
*** lucasagomes has joined #tripleo09:36
mordredjang: yay!09:38
mordredjang: so perhaps we should makea  new release?09:38
*** jp_at_hp has joined #tripleo09:38
*** shakamunyi has quit IRC09:38
*** nati_ueno has quit IRC09:38
*** bauzas has quit IRC09:39
lifelessGheRivero: I'm not sure. There should be.09:40
lifelessmordred: no, we have a crucial fix to do first.09:40
*** martyntaylor has quit IRC09:40
*** martyntaylor1 has joined #tripleo09:41
lifelessmordred: which will also fix the use of file:// for you :)09:41
lifelessmordred: I added you as reviewer on it09:41
*** lucasagomes has quit IRC09:41
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Install the check_mk Agent  https://review.openstack.org/8148509:44
*** e0ne_ has joined #tripleo09:46
*** nati_ueno has joined #tripleo09:47
*** e0ne has quit IRC09:49
*** Matt2 has quit IRC09:55
*** xuhaiwei has quit IRC10:00
*** lynxman has quit IRC10:04
*** CaptTofu has joined #tripleo10:05
*** lynxman has joined #tripleo10:05
*** jcoufal has quit IRC10:07
*** CaptTofu has quit IRC10:09
*** e0ne_ has quit IRC10:10
*** bauzas has joined #tripleo10:13
*** e0ne has joined #tripleo10:17
*** ramishra has quit IRC10:21
*** ramishra has joined #tripleo10:22
*** ramishra has quit IRC10:22
*** rpodolyaka1 has joined #tripleo10:25
*** rpodolyaka1 has quit IRC10:29
*** giulivo has joined #tripleo10:31
*** jistr|mobile has joined #tripleo10:31
*** vkozhukalov_ has joined #tripleo10:33
*** shakamunyi has joined #tripleo10:35
*** shakamunyi has quit IRC10:39
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Install the check_mk Agent  https://review.openstack.org/8148510:40
*** ramishra has joined #tripleo10:42
*** derekh has joined #tripleo10:54
openstackgerritMichael Kerrin proposed a change to openstack/diskimage-builder: WIP - enable elements to be re-playable  https://review.openstack.org/8149411:00
openstackgerritMichael Kerrin proposed a change to openstack/diskimage-builder: Add script to replay elements in an existing image  https://review.openstack.org/8026711:00
openstackgerritMichael Kerrin proposed a change to openstack/diskimage-builder: Add utility to mount and unmount images  https://review.openstack.org/7650911:00
openstackgerritMichael Kerrin proposed a change to openstack/tripleo-image-elements: WIP - enable elements to be replayable  https://review.openstack.org/8149511:03
*** nati_ueno has quit IRC11:06
*** ramishra has quit IRC11:07
*** jcoufal has joined #tripleo11:13
*** ccrouch has joined #tripleo11:15
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Better describe the compatibility mode being echo'd  https://review.openstack.org/8029611:20
*** e0ne has quit IRC11:21
*** matsuhashi has quit IRC11:22
*** matsuhashi has joined #tripleo11:24
*** rpodolyaka1 has joined #tripleo11:25
*** lucasagomes has joined #tripleo11:29
*** rpodolyaka1 has quit IRC11:29
*** fandi has quit IRC11:30
*** Matt2 has joined #tripleo11:30
*** shakamunyi has joined #tripleo11:36
*** matsuhashi has quit IRC11:37
StevenKderekh: Hm. I'd rather we focused on *removing* the compatability mode entirely11:39
derekhSpamapS: Fair enough, I was just improving a message I saw and had no idea what it meant11:39
StevenKderekh: I'm not sure if lifeless finished refreshing the test environment, but https://review.openstack.org/#/c/78467/ does so, and I have a local commit that polishes it off.11:39
*** shakamunyi has quit IRC11:40
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Remove backwards compatability in setup-baremetal  https://review.openstack.org/7846711:40
StevenKIf it passes CI, the testenv has been refreshed. I shall chase tomorrow.11:41
derekhStevenK: k11:43
StevenKOh, blah, it's missing a change to write-tripleorc11:47
lifelessderekh: I build a new testenv image and loaded it into glance in the hp region11:48
lifelessderekh: but CI was busy so I waited, then timed out to EOD :)11:48
derekhlifeless: ok, I don't suppose it included https://review.openstack.org/#/c/76171/311:50
*** CaptTofu has joined #tripleo11:50
openstackgerritSteve Kowalik proposed a change to openstack/tripleo-incubator: Remove backwards compatability in setup-baremetal  https://review.openstack.org/7846711:51
*** giulivo has quit IRC11:51
derekhlifeless: or https://review.openstack.org/#/c/79663/ (this one will be needed)11:51
*** e0ne has joined #tripleo11:52
derekhany cores out there could look at https://review.openstack.org/#/c/79663/ and https://review.openstack.org/#/c/76171/3 please11:52
Ngderekh: sure11:52
derekhand https://review.openstack.org/#/c/80603/11:52
* Ng between talks atm11:52
derekhNg: thanks11:52
Ngthis is not the biggest conference in the world ever :)11:52
*** lucasagomes has quit IRC11:53
*** jrist has quit IRC11:53
derekhNg: just a tad smaller then the openstack summit?11:53
Ngderekh: at least one order of magnitude smaller :)11:53
mordredlifeless: awesome. thanks!11:54
*** e0ne has quit IRC11:56
Ngderekh: done :)11:56
mordredlifeless, SpamapS, Ng: SpamapS patch to pypi-mirror seems to be unhappy on integration gate11:58
derekhNg: thanks12:02
*** al has quit IRC12:02
*** al has joined #tripleo12:03
*** hashar has quit IRC12:04
*** e0ne has joined #tripleo12:07
*** e0ne_ has joined #tripleo12:10
*** e0ne has quit IRC12:10
openstackgerritA change was merged to openstack-infra/tripleo-ci: Allow the testenv client to dictate the worker timeout  https://review.openstack.org/7617112:11
*** dprince has joined #tripleo12:11
*** bauzas has quit IRC12:12
openstackgerritA change was merged to openstack-infra/tripleo-ci: Install jq on Test Environment hosts  https://review.openstack.org/7966312:12
*** hashar has joined #tripleo12:14
derekhand another needed before building new test env hosts https://review.openstack.org/#/c/80265/12:15
derekhand a nice to have https://review.openstack.org/#/c/80603/112:16
*** bauzas has joined #tripleo12:17
*** jcoufal has quit IRC12:19
*** al has quit IRC12:24
*** al has joined #tripleo12:24
*** rpodolyaka1 has joined #tripleo12:26
*** e0ne has joined #tripleo12:28
*** rpodolyaka1 has quit IRC12:30
*** e0ne_ has quit IRC12:32
rcarrillocruzguys, i'm getting 'libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied' when 'configure-vm' is called from 'setup-seed-vm'12:35
rcarrillocruzshouldn't it be called with sudo?12:35
*** shakamunyi has joined #tripleo12:36
*** Killer_prince has quit IRC12:37
*** jtomasek has quit IRC12:37
*** weshay has joined #tripleo12:40
*** shakamunyi has quit IRC12:41
openstackgerritMichael Kerrin proposed a change to openstack/tripleo-image-elements: WIP - enable elements to be replayable  https://review.openstack.org/8149512:43
openstackgerritGonĂ©ri Le Bouder proposed a change to openstack/tripleo-incubator: Rest syntax fixes  https://review.openstack.org/8101212:43
*** bauzas has quit IRC12:44
*** bauzas has joined #tripleo12:44
*** mrunge has quit IRC12:51
*** jrist has joined #tripleo12:58
openstackgerritMichael Kerrin proposed a change to openstack/tripleo-incubator: Cause wait_for heat stack-create fail fast  https://review.openstack.org/7653712:59
*** lblanchard has joined #tripleo13:00
openstackgerritJon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Create pip manifest files of installed packages  https://review.openstack.org/8064413:09
openstackgerritJon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Make os-refresh-config install.d hook run later  https://review.openstack.org/8117413:09
*** hashar has quit IRC13:12
*** julim has joined #tripleo13:12
*** hashar has joined #tripleo13:14
*** nosnos has quit IRC13:24
*** rpodolyaka1 has joined #tripleo13:26
*** jrist has quit IRC13:27
*** rpodolyaka1 has quit IRC13:31
*** jrist has joined #tripleo13:36
*** shakamunyi has joined #tripleo13:37
*** funzo has quit IRC13:40
*** shakamunyi has quit IRC13:41
*** lucasagomes_ has joined #tripleo13:45
*** funzo has joined #tripleo13:50
*** nati_ueno has joined #tripleo13:50
*** jpeeler has quit IRC13:55
*** jpeeler has joined #tripleo13:56
*** nati_ueno has quit IRC14:03
*** jrist has quit IRC14:05
*** vkozhukalov_ has quit IRC14:05
*** lucasagomes_ has quit IRC14:06
*** vkozhukalov_ has joined #tripleo14:08
*** shakamunyi has joined #tripleo14:18
openstackgerritAlexis Lee proposed a change to openstack/tripleo-incubator: Use known dir instead of relying on BASH_SOURCE  https://review.openstack.org/8153914:22
*** rpodolyaka1 has joined #tripleo14:27
bnemecrcarrillocruz: You may need to log out and log back in.14:31
*** rpodolyaka1 has quit IRC14:31
bnemecrcarrillocruz: See https://bugs.launchpad.net/tripleo/+bug/128528514:31
uvirtbotLaunchpad bug 1285285 in tripleo "libvirtd group addition doesn't stick" [Medium,Fix released]14:32
bnemec(note that the bug status lies - the fix was reverted because CI somehow depends on this broken behavior)14:32
*** shakamunyi has quit IRC14:52
*** shakamunyi has joined #tripleo14:52
rcarrillocruzguys, is ubuntu 12.04 supported in tripleo? i found some mixed opiniones on this in email threads and such14:53
rcarrillocruzhttps://bugs.launchpad.net/tripleo/+bug/129293814:53
uvirtbotLaunchpad bug 1292938 in tripleo "dkms fails due to devtest not requiring linux-headers package as dependency" [Undecided,Triaged]14:53
rcarrillocruzi can't repro on 13.10, but i can on 12.0414:53
rcarrillocruzif we don't support it (or we are about to ditch support), then it's not worth working on a fix14:53
rcarrillocruzthanks bnemec, i just made it work randomly by closing/reopening my session after lunch, and now i just read your comment, hehe14:54
SpamapSmordred: ack14:55
*** jrist has joined #tripleo14:59
*** morazi has joined #tripleo15:01
openstackgerritAllison Randal proposed a change to openstack/tripleo-heat-templates: Pass undercloud metadata to overcloud compute  https://review.openstack.org/8155215:07
*** yamahata has joined #tripleo15:07
jp_at_hptchaypo: did you see that I updated https://review.openstack.org/#/c/81174/ to rebase it on slagles change?  I'm hoping that was OK?  I just wanted to get my updated patch up, as I was asked to rebase it onto your change.15:14
*** spzala has joined #tripleo15:15
SpamapSjp_at_hp: It is highly encouraged to "do the right thing" :)15:19
SpamapSjp_at_hp: A few months back we sent out some messages on the openstack-dev list that encouraged everyone to say "Thanks for fixing my patch." :)15:20
openstackgerritColeman Corrigan proposed a change to openstack/tripleo-incubator: use DIB_REPOLOCATION_ if set in pull-tools script  https://review.openstack.org/8155415:20
*** CaptTofu has quit IRC15:21
*** CaptTofu has joined #tripleo15:22
jp_at_hpSpamapS: it's just the unintended side-effects of bug reassignment, etc...15:23
*** CaptTofu has quit IRC15:26
*** rpodolyaka1 has joined #tripleo15:28
*** CaptTofu has joined #tripleo15:28
*** e0ne has quit IRC15:29
SpamapSjp_at_hp: Right, the machinery isn't perfectly oriented, but the community embraces it nonetheless.15:29
*** rpodolyaka1 has quit IRC15:32
*** rpodolyaka has quit IRC15:35
*** rpodolyaka has joined #tripleo15:36
*** bauzas1 has joined #tripleo15:38
*** bauzas has quit IRC15:38
*** jcoufal has joined #tripleo15:39
*** jcoufal has quit IRC15:39
*** petertoft has joined #tripleo15:42
*** bauzas1 has quit IRC15:42
*** bauzas has joined #tripleo15:42
*** newell has joined #tripleo15:45
*** lucasagomes_ has joined #tripleo15:48
*** e0ne has joined #tripleo15:53
andrearosagit status15:55
*** ccrouch has quit IRC16:02
openstackgerritAllison Randal proposed a change to openstack/tripleo-incubator: Pass undercloud metadata to overcloud compute  https://review.openstack.org/8157216:10
*** rlandy has quit IRC16:11
derekhcan I get some eyes on https://review.openstack.org/#/c/80265/ please ,16:12
derekhwe'll need that ^^ before creating a new TE image (https://review.openstack.org/#/c/80603/ would be nice too, but not needed)16:13
derekhlifeless: FYI, once that merges we can build/deploy an image16:14
*** noslzzp has quit IRC16:20
*** athomas has quit IRC16:21
bnemecslagle: derekh: https://review.openstack.org/#/c/74867/9 passed devtest against Fedora for me locally and it passed CI yesterday.16:21
*** lucasagomes_ has quit IRC16:21
bnemecThe dependent patch failed on a heisenbug, but since none of this stuff is actually tested in CI the test results there are kind of meaningless.16:22
*** dprince has quit IRC16:23
derekhbnemec: A+'d the one thats passed, agreed non of this is tested in the ci (and passed for me yesterday), sorry gotta wait on ci because its now a formality we need to follow, but will A+ once its done16:25
*** jrist has quit IRC16:25
*** morazi has quit IRC16:25
bnemecderekh: No problem, that's how it should be.  Thanks for looking.16:25
derekhbnemec: no prob16:26
jp_at_hpderekh: On https://review.openstack.org/#/c/80265/ - the commit message has about 3 grammar/spelling mistakes in it...16:26
derekhjp_at_hp: sounds about right, fixing, thanks16:26
jp_at_hpsorry - when it's just a commit emssage and not code I feel bad bringing it up :(16:27
jp_at_hpbecause I obviously can't type...16:27
bnemecMuphry's law strikes again!16:28
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Ensure json file is initialised  https://review.openstack.org/8026516:28
*** rpodolyaka1 has joined #tripleo16:28
*** bauzas has quit IRC16:28
derekhjp_at_hp: no prob, hope that's better.16:30
*** martyntaylor1 has quit IRC16:32
*** rpodolyaka1 has quit IRC16:32
*** UtahDave has joined #tripleo16:34
*** UtahDave has left #tripleo16:36
NobodyCamquestion on a odd seed vm error.. anyone encountering a strange error with rabbit saying it started but nothing listening on port 5672?16:38
*** hashar has quit IRC16:39
*** shakamunyi has quit IRC16:40
*** jistr|mobile has quit IRC16:42
*** sseago has quit IRC16:46
*** dprince has joined #tripleo16:47
*** sseago has joined #tripleo16:48
greghaynesI havent recently but there were 3 or so rabbit patches that merged yesterday16:49
clarkbSpamapS: pip shouldn't be case sensitive16:49
clarkbSpamapS: but if it is you can always use apache16:50
NobodyCamgreghaynes: ahhh16:50
*** epim has joined #tripleo16:51
NobodyCamgreghaynes: have you built a seed sense they landed?16:52
greghaynesNope, working on that now... ive run into a different bug while trying to build after recent merges16:52
greghaynesso hopefully soon16:52
NobodyCam:) greghaynes TY16:52
bnemecI just built a seed on Fedora this morning that I believe had the latest rabbitmq changes.16:53
bnemecDoesn't mean it's not broken on Ubuntu though.16:53
NobodyCambnemec: ack16:56
greghayneshuh, now I cant just run pull-tools16:58
bnemecgreghaynes: Do you have https://review.openstack.org/#/c/81443/ ?16:58
greghaynesGood call16:58
*** akuznetsov has quit IRC16:58
*** andreaf has quit IRC16:59
NobodyCamgreghaynes: bnemec: this is what I am seeing http://paste.openstack.org/show/YFnkjtyojkHUp2zDZqMF/17:00
*** markmc has quit IRC17:03
greghaynesWhats the problem thre?17:03
NobodyCamgreghaynes: note the netstat nothing on 567217:03
greghaynes.. huh17:05
greghayneswierd17:05
NobodyCamya17:07
bnemecFor comparison, here's what my working seed looks like: http://paste.openstack.org/show/73837/17:07
greghaynesmy devtest is spinning, shouldnt be much longer17:07
bnemecNot sure what the difference between beam and epmd17:07
greghaynesjust enough time for coffee \O/17:07
*** derekh has quit IRC17:07
NobodyCambnemec: ya line 26 shows rabbit listening17:08
bnemecHmm, mine is only listening on ipv6 5672.17:08
greghaynesI think that also means ipv417:08
NobodyCamya17:08
*** e0ne has quit IRC17:09
NobodyCambrb17:09
*** vkozhukalov_ has quit IRC17:14
bnemecrwsu: Is rabbitmq still broken with selinux?17:14
bnemecWondering if I can just fire off a devtest run with your mysql patch and expect it to work.17:14
greghaynesNobodyCam: My seed seems to of built fine on ubuntu17:15
*** morganfainberg_Z is now known as morganfainberg17:15
NobodyCamgreghaynes: :) humm17:16
rwsubnemec: yup still broken, runs veerry slow17:16
rwsubnemec: you'll run into other issues with selinux turned on even with mysql patch17:17
bnemecrwsu: Okay, thanks.  I won't try that then. :-)17:17
*** Lotus907efi_ has quit IRC17:17
*** ccrouch has joined #tripleo17:20
*** noslzzp has joined #tripleo17:29
*** rpodolyaka1 has joined #tripleo17:29
*** rpodolyaka1 has quit IRC17:33
jp_at_hpslagle: SpamapS on https://review.openstack.org/#/c/81174/ - should the package version of the install also move from 01 to 10 to maintain ordering in uses of the element that use different install types?17:40
*** weshay has quit IRC17:42
*** weshay has joined #tripleo17:42
*** sseago has quit IRC17:42
*** weshay has quit IRC17:43
*** weshay has joined #tripleo17:43
*** akuznetsov has joined #tripleo17:44
*** cwolferh_ has joined #tripleo18:02
*** cwolferh has quit IRC18:04
SpamapSjp_at_hp: well.. probably for consistency yes18:06
SpamapSjp_at_hp: but the move isn't necessary in that particular case because packages don't need pip ;)18:06
jp_at_hpSpamapS: not really my point.  Having them inconsistent could possibly introduce bugs seen only when using packages or only when using source, which would be a bad thing.18:07
*** epim has quit IRC18:09
*** petertoft has quit IRC18:09
NobodyCamgreghaynes: bnemec: this is a odd error: http://paste.openstack.org/show/T5dBmP0arbHOnC2iZgIn/18:09
*** rpodolyaka1 has joined #tripleo18:09
SpamapSjp_at_hp: Yes it could possibly be confusing I agree. However, using packages is more likely to introduce bugs that only happen when using packages.. so I see it as a low priority. ;) Good catch though!18:10
greghaynesNobodyCam: Interesting, check out your heat template18:10
greghaynesoh, this is for your seed?18:11
jp_at_hpSpamapS: I would place it as more important than grammar in a README though.18:11
NobodyCamgreghaynes: yep seed18:12
greghaynesNobodyCam: just for sanitys sake - have you rebased the patch your working on in t-i off master recently?18:14
greghaynesNot sure why the rabbit changes that went in to t-i-e would want that but *shrug*18:15
*** jtomasek has joined #tripleo18:15
bnemecNobodyCam: Could you have an older seed config.json?18:15
bnemechttps://github.com/openstack/tripleo-image-elements/blob/master/elements/rabbitmq-server/os-config-applier/mnt/state/var/lib/rabbitmq/.erlang.cookie#L118:15
bnemecI assume that needs to be set in it for this to work.18:16
bnemecI bet you're missing https://github.com/openstack/tripleo-image-elements/blob/master/elements/seed-stack-config/config.json#L1618:18
NobodyCambnemec: I'll bet your correct TY testing now18:20
slaglejp_at_hp: yea, i've kept the numbering the same, just for consistency really18:31
jp_at_hpthanks slagle18:32
slaglethe other aspect of it is if there are common install scripts that are just under install.d, you might want that to always run before (or after) the implementation under *-{package,source}-install18:32
jp_at_hpNow if only there were some cores to review ;-)18:32
SpamapSjp_at_hp: why do we have to assign a value to either thing? They should both be corrected.18:33
jp_at_hpgood point slagle - that's similar to what I was thinking of in other elements being dependent upon os-refresh-config.  That's definitely a more up=close and personalexample of why keeping them the same would be good.18:33
*** andreaf has joined #tripleo18:33
jp_at_hpSpamapS: and lo - they are :D18:34
*** sseago has joined #tripleo18:37
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: DHCP: make udev rules want dhcp-interface@.service  https://review.openstack.org/8159618:42
NobodyCambnemec: greghaynes: TY seems to have gotten me past that issue... :)18:42
bnemec\o/18:43
*** spzala has quit IRC18:44
greghaynesw00t, that means we get ironic support now, right?18:44
greghaynes;)18:44
NobodyCam:) we'll see attempting to deployundercloud via ironic.18:45
NobodyCamright now18:45
*** akuznetsov has quit IRC18:45
*** shakayumi has joined #tripleo18:47
*** shakayumi has quit IRC18:47
*** julim has quit IRC18:47
*** saurabhs has joined #tripleo18:57
*** jp_at_hp has quit IRC18:58
*** shakayumi has joined #tripleo18:58
*** noslzzp has quit IRC19:01
*** shakayumi has joined #tripleo19:02
*** derekh has joined #tripleo19:02
*** noslzzp has joined #tripleo19:03
saurabhslifeless: for trove disk image builder elements, I am getting a warning 'WARNING: first-boot.d is DEPRECATED, it will be removed rom diskimage-builder in a future release' can you tell me what should be a replacement for first-boot.d going forward19:03
lifelesssaurabhs: idempotent os-refresh-config scripts19:03
*** shakayumi has joined #tripleo19:04
derekhlooks like our geard fell over again, restarted now currently running ci jobs will probably fail, but should be ok after that19:04
*** e0ne has joined #tripleo19:04
clarkbits interesting that your geard falls over a lot19:04
clarkbthe logstash geard and zuul geard seem mostly solid19:04
NobodyCamundercloud | CREATE_IN_PROGRESS & undercloud-undercloud-vtdzoh2xs7ai | BUILD  | spawning lloking good so far!19:05
derekhclarkb: its because we weren't closing connections to it, we have to build new test environments with the fix and deploy them19:06
clarkboh right you go through objects19:06
saurabhslifeless: os-refresh-config runs everytime there is metatata change. firstboot.d runs only when machine boots up for the first time. are they both different? or is there something in os-refresh-config that runs only on first boot19:07
derekhclarkb: yup, and eventually hit error: [Errno 24] Too many open files19:07
derekhlifeless: before we can deploy new testenv's we need https://review.openstack.org/#/c/80265/ (and ideally a new release of geard the new release is waiting on https://review.openstack.org/#/c/80304/)19:09
SpamapSsaurabhs: if [ -e /some/state/file ] ; then exit 0 ; fi ... do idempotent first boot thing here ... ; touch /some/state/file19:10
lifelessderekh: i19:11
lifelessderekh: huh, where are we passing an absent file in ?19:14
derekhlifeless: https://github.com/openstack-infra/tripleo-ci/blob/master/elements/testenv-worker/bin/ensure-test-env#L3319:15
lifelessderekh: oh, I see. So I want us to actually call into devtest_testenv.sh there. Hmm, I guess the handle missing path is ok19:16
derekhlifeless: yup, agreed, I intended to do that but keep getting pulled onto fixing what we currently have19:17
NobodyCamgreghaynes: bnemec: undercloud deployed from seed via Ironic using USE_IRONIC=1; devtest.sh ... patches to follow after lunch19:17
greghaynes\O/19:17
lifelessderekh: ok, so land that, because we don't CI create-nodes, it can't possibly affect CI one way or another19:18
NobodyCam:)19:18
greghaynespatches or it didnt happen :p19:18
lifelessSpamapS: https://review.openstack.org/#/c/80265/ ^19:18
*** sseago has quit IRC19:18
NobodyCamgreghaynes: lunch first !19:19
NobodyCam:-p19:19
*** shakayumi has quit IRC19:19
SpamapSderekh: https://review.openstack.org/#/c/80265/ -1'd for commit message .. fix quick pls will review again now19:19
NobodyCamgreghaynes: http://paste.openstack.org/show/vykcN27wlPuhrrVV3aov/19:20
greghaynesnoice19:20
NobodyCam:)19:20
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Ensure json file is initialised  https://review.openstack.org/8026519:21
lifelessdevananda: hi19:21
lifelessactually nvm19:21
derekhSpamapS: thanks, done19:21
*** rpodolyaka1 has quit IRC19:22
*** spzala has joined #tripleo19:23
*** vkozhukalov_ has joined #tripleo19:23
*** rpodolyaka1 has joined #tripleo19:25
*** spzala has quit IRC19:26
derekhlifeless: so I've been running a script that I think gives a nice overview of how our jobs are doing http://goodsquishy.com/downloads/tripleo-jobs.html , I've found it very handy to monitor in order to quickly identify problems, needs a more permanent home though19:27
lifelessderekh: elastic-rechecj ? :)19:28
*** Lotus907efi has joined #tripleo19:28
*** petertoft has joined #tripleo19:29
derekhlifeless: ya, I've been toying with elasic recheck aswell, but will it tell us things like "none of our overcloud jobs has passed since 2pm yesterday" , these things are going unnoticed until people spend time looking into why ci failed on their last 3 attempts by which time a day has passed19:30
derekhlifeless: would elastic recheck only help if we add queries for known intermittent failures19:31
lifelessjogo: ^19:31
derekhlifeless: Of which we currently have one http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZVwiQ2Fubm90IGZldGNoIGluZGV4IGJhc2UgVVJMXCIgQU5EIGJ1aWxkX25hbWU6Y2hlY2stdHJpcGxlby0qLXByZWNpc2UiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTUxNjUxMzY0NjksIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=19:31
derekhlifeless: I can add that queries to elastic recheck for tripleo jobs which will help in that instance19:32
NobodyCamTripleO element question, would it be better to rename and create new elements for seed-stack-config.. ie. seed-stack-config-nova & seed-stack-config-ironic.. or keep single seed-stack-config element?19:32
lifelessNobodyCam: single one probably19:34
NobodyCamlifeless: ack :)19:35
greghayneslifeless: So I think the galera 'works' now with 5.5, I must of been trying to debug 5.6 without realizing when it was OOMing19:36
greghaynes\O/ galera might actually merge now19:36
greghaynesalso, all tests seemed to of passed19:36
SpamapSgreghaynes: s/must of/must have/ (and yes, I will correct it every time I see it)19:38
SpamapSbecause I am working on transitioning to full grumpy old man status19:38
greghaynesargh, and I thought I was getting better19:38
*** julim has joined #tripleo19:38
greghaynesalso, seem to have passed19:39
SpamapSConverting Heat templates to softwareconfig is helping quite a bit with the "get grumpy" part19:39
greghayneshaha19:39
*** fandi has joined #tripleo19:41
*** petertoft has quit IRC19:43
openstackgerritA change was merged to openstack/tripleo-incubator: Ensure json file is initialised  https://review.openstack.org/8026519:44
*** hashar has joined #tripleo19:47
derekhlifeless: submitted recheck query for that pip bug https://review.openstack.org/#/c/81609/19:48
lifelessgreghaynes: wicked19:52
derekhbtw, looks like the root of that intermittent pip error seems to be caused by pip not receiving a response from the server before a 15 second timeout (digging into pip code now to figure out if there is a way around it)19:57
lifelessderekh: yes, this is why I want local mirrors of stuff19:58
lifelessderekh: we have regular network blips, and we don't know why19:58
derekhwe only see the problem in packages where cases mismatch because pip tries to get the URL 2 ways, when there is a case mismatch we have no failabck and are mor likely to fail19:59
Lotus907efiI have another person here who has a SL390 with 24 cores and 94GB of memory and she is trying to run devtest.sh using the pypi internal mirror I have setup as well as the internal Ubuntu saucy repo20:00
Lotus907efibut her devtest.sh run is taking 6+ hours20:00
Lotus907efiwhich is way too long20:01
derekhin the case I managed to catch with tcpdump, it took 21 seconds before we got a response from server20:01
Lotus907efiI am looking at her output file and the thing looks like it is using the mirrors20:01
derekhLotus907efi: the only time I saw really long times like that kvm wasn't loaded, sudo lsmod | grep -i kvm (allthough if your time is spent building images then thats not the problem)20:03
Lotus907efihmmm20:04
lifelessLotus907efi: where is the time going - image builds or vm deploys ?20:06
*** epim has joined #tripleo20:06
Lotus907efiI am trying to examine the output file I have to see what the heck is happening20:06
*** CaptTofu has quit IRC20:06
lifelessLotus907efi: can you pastebin it? or something?20:07
Lotus907efiummm, it is 2.8MB20:07
Lotus907efilet me see if I can figure out a way to do that20:08
Lotus907efiI could attach it as a file to a bug :)20:08
lifelessyou can pastebin 2.8MB20:08
lifelesspastebinit < file.name20:09
Lotus907efia bug for "devtest takes WAY TOO LONG!"  :)20:09
Lotus907efiah, ok20:09
openstackgerritJames Slagle proposed a change to openstack/tripleo-image-elements: Always create o-a-c's $TEMPLATE_ROOT  https://review.openstack.org/8161420:09
SpamapSThere's a little program on Ubuntu..20:09
SpamapSkvm-ok20:09
lifelessderekh: ok so do you want me to deploy a new testenv, or will you ?20:12
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Add cinder-lio element  https://review.openstack.org/7846320:13
openstackgerritBen Nemec proposed a change to openstack/tripleo-image-elements: Factor out tgt-specific parts of cinder element  https://review.openstack.org/7846220:13
derekhlifeless: I'll kick off a image build now, on train at the moment so not sure how far I'll get, will let you know20:14
lifelessderekh: ack ack20:14
Lotus907efihmm, I tried to pastebin the log and it said:  Bad API request, maximum paste file size exceeded20:16
lifelessLotus907efi: hah nuts :)20:16
lifelessLotus907efi: upload it to swift?20:16
lifelessI hear we have this cloud thing20:17
Lotus907efihmmm, I don't have access to any swift datastores20:17
Lotus907efiso submit a dummy bug and then attach the log to that?20:18
Lotus907efilifeless: are you or greg on hipchat?20:19
lifelessoften20:19
lifelessLotus907efi: I'm in the TripleO room20:20
*** rpodolyaka1 has quit IRC20:20
openstackgerritDan Prince proposed a change to openstack/diskimage-builder: serial-console: Use udev rules to startup getty  https://review.openstack.org/8161620:23
*** rpodolyaka1 has joined #tripleo20:24
tchaypoLotus907efi: my first run on a desktop -grade machine took longer than that, but since I set up squid and the pypi-mirror element (the one that caches things in ~/.cache/image-create/pypi, not a real pypi mirror) my times have dropped right down20:24
Lotus907efihmmm20:24
Lotus907efiwell I would hardly call a 24 core system with 94GB of memory a desktop grade system :)20:24
Lotus907efior at least I wouldn't mide having it as a desktop system20:25
Lotus907efis/mide/mind20:25
*** rpodolyaka1 has quit IRC20:28
*** Meena has joined #tripleo20:29
tchaypoyeah, I'd expect it to be able to cope better20:30
tchaypoalthough, from what I've seen, my system doesn't seem to be CPU-bound20:31
jogoderekh: that is correct20:31
tchaypothe first few runs it was network-bound; once it had most things cached it seems to spend most of its time building disk images20:31
*** CaptTofu has joined #tripleo20:32
jogoderekh: so the basic log workflow that gate uses is:20:33
jogodevstack-gate runs tempest against cloud and we collect logs20:33
jogologs (tempest,openstack, etc)  are stored in file server  and a subset are pushed into a massive elsticsearch cluster20:33
jogoelastic-recheck is a collection of elasticsearch fingerprints for bugs20:34
jogoex:  if you see this stacktrace then you hit bug x.20:34
jogoelastic-recheck is used for two things:20:34
jogolisten to gerrit stream and help developers figure out if there job failed due to a known bug20:35
jogoand secondly help us track race conditions and bugs20:35
jogovia http://status.openstack.org/elastic-recheck/ and  http://status.openstack.org/elastic-recheck/data/uncategorized.html20:35
derekhjogo: yup and in order for us to get feedback from elastic recheck we need to configure it with known fingerprint e.g. https://review.openstack.org/#/c/81609/20:36
lifelessderekh: so, I like your script, but think it should build on es :)20:36
jogowell thats assuming you have all the data in a elasticsearch cluster20:36
derekhlifeless so basically we need something to alert us to the fact that ci is failing all around us and its not obvious until somebody sees enough failures to dig in a take a look at why, e.g. the nova issue that last hit us was a problem for 4 days (although 2 were weekend)20:37
jogoderekh: so I think we have/had that bug as a fingerprint already20:37
*** dprince has quit IRC20:37
jogoderekh: we have been using http://status.openstack.org/elastic-recheck/data/uncategorized.html for that in openstack20:37
jogothat is a list of unclassified gate failures. gate because we don't expect gate to fail ... versus check20:38
Lotus907efihmmm, the django-openstack-auth python module seems to require a Django version newer than 1.4 and less than 1.6 but the run-mirror pulled in only 1.6.2 which did not satisfy the requirement20:38
jogoderekh: and its working pretty well for us20:39
derekhjogo: why isn't it reporting back to tripleo reviews? because it a defferent queue?20:39
derekh*different20:40
jogoderekh: not sure, right now its not reporting because ES is backed up20:40
tchaypolifeless: if we fix the pip problem by having a local pypi mirror, doesn't that mean we'd have to tell anyone who wants to deploy with our tooling that setting up a 70Gb pypi mirror locally is a precondition for a working deploy?20:40
jogoway up20:40
derekhjogo: ok20:40
jogoqueries/1272417.yaml20:41
jogois the bug you just filed20:41
derekh1292141.yaml20:42
jogoderekh: yours is a duplicate20:43
*** jang1 has joined #tripleo20:43
jogois there a gate-tripleo queue?20:44
jogoor just check20:44
derekhjogo: ahh ok, just check20:45
derekhjogo: check-tripleo20:45
derekhlifeless: image built -rw-r--r-- 1 root root 619553792 Mar 19 20:38 /opt/stack/testenv-worker.qcow220:47
greghaynestchaypo: pypi-mirror seems to work with only mirroring things we need20:48
jogoderekh: in that case the easiest way to figure out of  things are down is graphite20:48
greghaynestchaypo: mine is 193M20:48
derekhlifeless: I'm happy to test it later before we deploy system wide, but wont be able to for a few hours as I'll be dropping off soon20:48
derekhlifeless: or we could just deploy and keep our fingers crossed20:48
jang1quite pleased with the latest pypi-mirror, by the way. Seems to iron out a lot of the bugs.20:48
Lotus907efiwhat was the env variable to set that would keep the cache?20:49
greghaynesUSE_CACHE20:49
bnemecYou have to pass -c to devtest.sh20:49
greghaynesits not 'keep the cache'20:49
bnemecIt resets USE_CACHE to 020:49
Lotus907efiok, so just set that to 1 and export before running devtest ?20:49
Lotus907efioh20:49
Lotus907efiok20:49
greghaynesjust if it finds the image on disk it uses it rather than rebuilding20:49
derekhjogo: ahh is graphite running somewhere ?20:49
* bnemec just made that mistake :-)20:49
tchaypogreghaynes: but if I've understood correctly, lifeless is suggesting we need to have a full pypi mirror local to our test env to get reliable CI20:50
jang1^^ that really, really helps20:50
jang1by "full" - pypi-mirror makes one. You don't need the 80G bandersnatch variant20:50
jogoderekh: yup http://graphite.openstack.org/20:51
greghaynestchaypo: Id be surprised if full really means 'all the packages'20:51
jogoand you will want something like stats.zuul.pipeline.check-tripleo.job.check-tripleo-undercloud-precise.FAILURE20:51
greghaynesbecause... you shouldnt need that20:51
greghaynesjust all the packages we use20:51
derekhjogo: cool, thanks I'll give it a whirl and see what happens20:52
tchaypoThe problems being talked about above where pip failing to get the index because it times out after 15 seconds. afaik pypi-mirror doesn't create an index.20:52
jogoyou can do fancy graphite tricks and get failure percentege20:52
jogoand do something like this: jogo.github.io/gate'20:52
jogojogo.github.io/gate20:52
tchaypobut maybe the solution is that pypi-mirror should locally cache an index and we pass that to pip?20:52
Lotus907efimy mirror here is only 330MB20:52
*** rpodolyaka1 has joined #tripleo20:53
*** e0ne has quit IRC20:53
jang1have you got everything up and working now, Lotus907efi?20:53
Lotus907efimy mirror is being referenced20:53
*** vkozhukalov_ has quit IRC20:54
*** e0ne has joined #tripleo20:54
Lotus907efionce in a while it points up missing packages though20:54
Lotus907efiI just had an issue with a version problem on Django package20:54
Lotus907efiand had to go get an older version20:54
Lotus907efito meet the requirements of a package that was referencing Django20:55
jang1rather than suffer the death of a thousand cuts over the next n weeks, I encouraged everyone in the office to front-load the pestering for help over the last couple of days.20:56
jang1my god, but we've seen some bizarre things.20:56
jang1my favourite was the devtest_seed run that just stopped dead (bash hung) after build-seed-vm.sh returned.20:56
jang1I have honestly no idea _why_ it did that, although we eventually caught an errant ssh process that seemed to trigger it.20:57
*** rpodolyaka1 has quit IRC20:57
*** e0ne has quit IRC20:57
jang1also occasionally seeing tmpfses that aren't being cleaned up - looks like various system processes get their cwds into it.20:57
SpamapSjang1: there are some tmpfs leaks20:58
SpamapStoday I feel like there is a brain leak.. can't seem to get my head focused20:58
*** epim has quit IRC20:58
*** nati_ueno has joined #tripleo21:01
openstackgerritChris Krelle proposed a change to openstack/tripleo-image-elements: Add Ironic config.json to seed-stack-config element  https://review.openstack.org/8162721:01
tchaypojang1: was that your work on pull-tools that I patched over yesterday?21:01
tchaypojang1: Does my patch still do what you intended?21:02
*** epim has joined #tripleo21:03
jogoderekh: http://jogo.github.io/gate/tripleo.html21:03
lifelessjang1: re errant ssh process21:03
jogocheck data is very very dirty though21:03
jogobut that gives you an idea21:04
lifelessjang1: so ssh won't disconnect if it's fds are not closed21:04
* tchaypo facepalms21:04
openstackgerritChris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic bits to _undercloud  https://review.openstack.org/7296921:04
jang1that wasn't the problem here. literally, bash hung as boot-seed-vm exited and didnt' get as far as assigning to SEED_IP.21:04
jang1I have honestly never seen anything like ti21:05
tchaypoGoogle calendar has a timezone labelled "(GMT +00:00) GMT (no daylight savings)"21:05
lifelessjang1: so you ssh to a host and run foo, foo gets stdout back to ssh, if foo blocks for some reason then the ssh process will hang21:05
lifelessjang1: ok21:05
tchaypoI think that possibly means "UTC", which is sometimes one hour different from GMT?21:05
jang1GMT is UTC (only not sidereal, or what have you)21:06
jang1british DST isn't called GMT21:07
derekhjogo: thanks, need to do some graphite research21:07
NobodyCamgreghaynes: i'm waiting to see if the new https://review.openstack.org/#/c/72969 takes care of the key issue21:07
derekhtchaypo: I thought GMT and UTC were always within seconds of each other21:07
jogoyou can see the query I used if you copy the image url21:08
*** lazy_prince has joined #tripleo21:08
*** lazy_prince is now known as killer_prince21:08
jogoand replace /render/ with /graphlot/21:08
jogoor or just /plot/21:09
tchaypodamn jang1 and derekh, you've spoiled my rant.21:09
tchaypohrmph.21:09
tchaypoFortunately this means I can move the target of my rant to the timezone labelled "(GMT +00:00) London"21:10
tchaypowhich is self-contradictory, because London isn't always GMT +021:10
tchaypoI always feel those labels should either say "London" or "GMT+0", putting both in is confusing.21:11
tchaypoespecially when you have places like eastern australia, where Google lists "GMT+10 Brisbane", "GMT+10 Melbourne/Sydney" and "GMT+10 Hobart"21:11
*** martyntaylor has joined #tripleo21:12
*** blamar is now known as blamar-away21:12
* tchaypo drinks more coffee21:12
Lotus907efihow long should a undercloud create process take?21:12
Lotus907efiI am at:  Waiting for the undercloud stack to be ready21:13
Lotus907efi+ wait_for 220 10 stack-ready undercloud21:13
jang1the DIB part or the heat part?21:13
tchaypoLotus907efi: I find that the check there isn't particularly good at catching failures. I usually jump onto the seed and keep an eye on "heat event-list undercloud" so I find out about failures early.21:13
Lotus907efiheat stack-list says the status is "CREATE_IN_PROCESS"21:13
jang1it takes "a while" - you can log onto the seed and see where it's got to if you want. Or even log into the undercloud, once it's up, and watch the progress from within21:13
jang1I think the undercloud standup (heat aprt) took about 300-350 seconds on my desktop today21:14
*** hashar has quit IRC21:14
*** lblanchard has quit IRC21:14
Lotus907efihmmm21:15
Lotus907efiheat event-list undercloud is interesting21:15
jang1what about nova list? is the undercloud actually active yet?21:15
Lotus907efione small problem the person who first setup this system and environment put a passphrase on the ssh key21:16
Lotus907efiso once in a while it sits there waiting for the damn passphrase to be input21:16
Lotus907efiso I don't know if that is the problem now or not21:16
Lotus907efiundercloud is ACTIVE21:17
Lotus907efi(it was sitting there waiting for the passphrase) ARGH!21:17
clarkbreplace the key?21:18
Lotus907efiyeah, but if I do that will I have to start from scratch again?21:20
*** noslzzp has quit IRC21:21
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Set +x on executable files  https://review.openstack.org/8163321:22
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Add dib-lint script  https://review.openstack.org/8163421:22
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Alphabetize all element-deps  https://review.openstack.org/8163521:22
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Enforce alphabetical ordering of element-deps  https://review.openstack.org/8163621:22
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: WIP: Make sure all scripts are set -e  https://review.openstack.org/8163721:22
clarkbI wouldn't expect so if you strip the passphrase off the key21:22
clarkbthat way all of the public keys will still match21:22
clarkbbut I am not familiar enough with devtest to know for sure21:22
jang1it should certainly work, yeah21:22
tchaypossh-keygen -N will let you change the passphrase21:22
tchayposo you shouldn't need to change anything since you'll still be using the same key21:23
tchaypoas clarkb already said21:23
derekhlifeless: new image in /opt/stack/testenv-worker.qcow2 I'll be back later to test it if (unless you deploy it first)21:23
*** derekh has quit IRC21:24
bnemecAre rechecks going to work right now or do we need to wait for the new testenv derekh was working on?21:31
lifelessbnemec: CI should be up right now21:36
lifelessI have a dr appt and will after that tear things down and redeploy the testenvs21:36
bnemecCool.  Didn't want to fire off rechecks if they were just going to fail again.21:36
bnemecThanks21:36
lifelessbnemec: why did they fail ?21:36
*** lifeless changes topic to "tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt"21:37
bnemeclifeless: Not sure, but something appears to have been hosed in the test environment: http://logs.openstack.org/66/74866/9/check-tripleo/check-tripleo-seed-precise/a89e2e5/console.html21:37
lifelessLotus907efi: wow the ps -fH's really make that log hard to read21:38
Lotus907efioh?21:38
lifelessLotus907efi: yeah, its about 80% ps -fH output :(21:38
Lotus907efiah21:38
Lotus907efiis that normal for that to be in there so much?21:38
lifelesswe don't do that at all21:39
lifelessit must be a thing your colleague did21:39
lifeless"Michey ps -fH"21:39
Lotus907efihmmm21:39
Lotus907efiweird21:39
lifelessok so this:21:40
lifeless10-os-collect-config                        531.88121:40
lifelessthats kindof long21:40
lifelessand21:40
lifeless01-os-refresh-config                        153.11421:40
lifelessand wow21:41
lifeless76-neutron                                 1201.21521:41
lifelessso this looks like either IO performance on the local machine, or network performance, to me21:41
lifelesswhere is there machine, relative to yours21:41
lifelessto your mirror, I mean21:41
lifelessLotus907efi: ^21:42
lifelessLotus907efi: I have to run to a dr appt, but I am speculating the answer is 'half way around the world' or something :)21:42
Lotus907efiit is in the same DC but in a different rack21:42
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Alphabetize all element-deps  https://review.openstack.org/8163521:42
Lotus907efiboth are in USeast21:42
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: WIP: Make sure all scripts are set -e  https://review.openstack.org/8163721:42
openstackgerritBen Nemec proposed a change to openstack/diskimage-builder: Enforce alphabetical ordering of element-deps  https://review.openstack.org/8163621:42
lifelessthis is what I'd expect to see:21:42
lifeless76-neutron                                   17.58221:42
Lotus907efihmmm21:42
lifelessgotta bounce, back in ~ 1 hr21:43
* bnemec doesn't know his abc's21:43
Lotus907efiwhere are you seeing those figures?21:43
Lotus907efitht is something I can grep for in the logs?21:43
jang1the profiling output, I'd expect.21:43
Lotus907efihmmm21:43
jang1dib dumps the time each script snippet took21:44
Lotus907efiI would like to compare those figures with the ones from my system that took a LOT less time21:44
Lotus907efiok21:44
Lotus907efihmm, my times seem a LOT less21:45
jang1so there's either a network problem - trouble talking to a proxy, perhaps.21:45
*** jtomasek has quit IRC21:45
jang1or, there's something else eating up all the disk bandwidth.21:45
Lotus907efilike 1/2521:46
Lotus907efihmmm21:46
jang1I don't think pip behaves particularly well in the face of dropped connections (it's not marvellously robust in that respect)21:46
Lotus907efifor instance my time for 10-os-collect-config  was on the order of 20.066 while her's was 531.88121:46
Lotus907efithat is a huge diff21:46
openstackgerritChris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic in to seed vm  https://review.openstack.org/8164321:46
jang1so if the network's behaving oddly, it's not behaving oddly enough to cause pip to error out21:47
jang1ooh ^^ that's an interesting change21:47
*** ccrouch has quit IRC21:51
*** rpodolyaka1 has joined #tripleo21:53
Lotus907efiyeah, is ironic far enough along to do that?21:54
jang1it's further along than I'd had the impression it was prior to the mid-session hackathon.21:55
jang1and, there's no better way to iron out the bugs than to use it in anger21:55
*** rpodolyaka1 has quit IRC21:57
Lotus907efitrue21:59
*** epim has quit IRC21:59
bnemecChris said he was able to successfully run devtest earlier today with Ironic.22:01
*** epim has joined #tripleo22:02
*** e0ne has joined #tripleo22:04
*** e0ne has quit IRC22:08
*** martyntaylor has quit IRC22:08
*** epim has quit IRC22:09
openstackgerritJenkins proposed a change to openstack/os-collect-config: Updated from global requirements  https://review.openstack.org/8164822:09
openstackgerritChris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic in to seed vm  https://review.openstack.org/8164322:13
*** spzala has joined #tripleo22:13
Lotus907efiif I have a devtest run that completed on a system successfully so the seed, undercloud, and overcloud VMs are all up, if I want to do another run while preserving the cache (using -c option) will the earlier VMs be shutdown and cleaned up properly?22:15
jang1yes22:15
NobodyCambnemec: yep I was able to use Ironic to deploy undercloud from seed22:19
*** bauzas has joined #tripleo22:19
tchaypoLotus907efi: If you're going to do it with devtest.sh make sure you pass --existing-environment22:23
tchaypothat will make it not skip _testenv.sh and just use the existing VMs22:23
tchaypo_testenv.sh performs poorly if there are already existing VMs - it doesn't create them, then it tramples on $TE_DATAFILE with its new information about not having any VMs22:24
tchaypothe alternative is to run cleanup-env, which will remove the VMs and make it safe to run _testenv.sh again22:25
openstackgerritChris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic in to seed vm  https://review.openstack.org/8164322:25
Lotus907efilifeless: did you ever get a reply from Rocky about the jq issue?22:27
*** fandi has quit IRC22:28
*** jang1 has quit IRC22:30
*** howleyt has joined #tripleo22:33
Lotus907efihmmm22:35
Lotus907efibut doesn't cleanup-env also remove the cached images?22:35
tchaypodunno *reads source*22:37
tchaypodoesn't look like it.22:37
tchaypovirsh destroy, virsh undefine, virsh vol-delete is all I'm seeing22:37
*** epim has joined #tripleo22:43
*** CaptTofu has quit IRC22:49
*** CaptTofu has joined #tripleo22:50
*** yfaykya has joined #tripleo22:51
lifelessLotus907efi: no reply so far22:51
Lotus907efihmmm22:52
lifelessLotus907efi: yes. look for PROFILING22:52
Lotus907efiI can walk over to building 6 again and needle him22:52
lifelessjang: Lotus907efi: note that DIB tries to build in a tmpfs if possible to avoid all disk write IO issues (syncs, seeks etc)22:52
Lotus907efiah, ok22:52
Lotus907efilifeless: did you see my reply that this system and my pypi mirror system are in USeast but separate racks?22:53
lifelessLotus907efi: I did22:54
*** CaptTofu has quit IRC22:54
howleytHi all22:54
*** rpodolyaka1 has joined #tripleo22:54
lifelessLotus907efi: does it have a distro mirror as well to use ?22:55
Lotus907efiyes22:55
lifelessok22:55
yfaykyahowleyt: yo22:55
lifelesslooking through the log for dib-run-parts Wed Mar 19 00:31:19 UTC 2014 Running /tmp/in_target.d/install.d/76-neutron22:55
Lotus907efithose accesses are pretty fast22:55
lifelessin more detail22:55
Lotus907efiI am watching the outout now22:55
Lotus907efioutput22:55
Lotus907efibut the access to my pypi mirror seem slower22:55
howleytI want to try bringing up a second notCompute node in my overcloud. What's the easiest way to do this?22:56
lifelesshowleyt: you'll need to change the overcloud-source.yaml, I *think* greghaynes might have a draft of such changes around22:57
howleytIs it a case of modifying overcloud-source.yaml to duplicate...22:57
lifelesshowleyt: no, you'll want a SCALE variable approach like we do for compute22:57
lifelesshowleyt: see Makefile22:57
greghayneshrm, I just copied the notcompute0 and s/notcompute0/notcompute1 in the destination block essentially22:58
howleytlifeless: that sounds better22:58
lifelesshowleyt: we've done some ground work for scaling notCompute but it was/is blocked on having HA ready plumbing22:58
lifelessgreghaynes: oh you bad boy :)22:58
greghayneshaha22:58
howleytYa, I started doing that and it felt bad..22:58
greghaynesYes, not sure its something I should really share22:58
tchaypoPlease share22:59
*** rpodolyaka1 has quit IRC22:59
tchaypoIt'll make the rest of us feel better about ourselves :22:59
tchaypo:)22:59
greghaynesI should probably mess with merge.py'ifying it now that I kind of understand whats going on....23:00
tchaypoi hate the "notCompute" name.23:02
howleytSo the merge script wouldn't necessarily handle a --scale NotCompute as it stands?23:02
greghaynestchaypo: So say we all23:02
greghayneshowleyt: it certainly doesnt23:02
greghayneshowleyt: pretty easy to test that :)23:02
howleytindeed (on the naming)23:03
*** yamahata has quit IRC23:04
greghaynesI wonder why we dont call it control... seems like we use that name elsewhere23:05
greghaynesor maybe thats exactly why we dont call it control23:05
howleytgreghaynes: I guess working on a copy as you did could work, for starters. Just wanted to check in case I was missing something.23:05
SpamapSgraaahhh.. why does vim insist on over-indenting yaml whenever I type something: ?23:06
greghayneshowleyt: yes, the issue is that running multiple nodes doesnt work as is not because of the template but because of rabbit and mysql not being able to handle it23:06
greghaynesas well as some state needing to be moved into the db AIUI23:07
tchaypogreghaynes: or rather, because we're not supplying configs to rabbit/mysql to tell them about the fact that they live in a cluster23:07
tchayporight?23:07
*** nati_ueno has quit IRC23:07
howleytSo I'm interested in trying out the rabbit cluster element changes that have landed.23:07
greghaynesthat might be true for rabbit after yesterdays patches, not true for mysql yet23:07
lifelessLotus907efi: so we need more timestamps from e.g. pip. but yes, we have ok mirror perf - Fetched 339 kB in 0s (1681 kB/s)23:08
lifelessfor linux packages23:08
tchaypoSpamapS: because you don't have filetype-specific ts/sw settings?23:08
howleytgreghaynes: understood23:08
Lotus907efihmmm, ok23:08
*** bauzas has quit IRC23:08
greghayneshowleyt: jprovazn is the person to chat with on that. You might be able to bake up a template that only clusters rabbit on the !0 notcompute nodes...23:08
SpamapStchaypo: it does it _while I type_23:08
Lotus907efihow to get more time stamps from pip?23:09
SpamapStchaypo: I presume this is 'autoindent' .. but I set noautoindent.. so... graahh23:09
howleytgreghaynes: Will do. So I'd have to make sure that mysql is only deployed on notcompute0 so that everything else works happily?23:10
greghaynesMy inexperienced mind thinks that might kind of work... I am still figuring this out as well though23:11
lifelessLotus907efi: not sure :)23:11
NobodyCamlifeless: just checking befor I start this is the in progress work I need to rebase off of: https://review.openstack.org/#/c/7846723:11
* greghaynes looks around for others to confirm/deny that idea23:11
*** newell has quit IRC23:11
lifelessNobodyCam: I think so yes. StevenK - I'm suggesting NobodyCam work with your nodes refactoring stuff23:11
tchaypohowleyt: that sounds feasbile.. but if mysql is only running on one node, why are you running up multiple nodes?23:12
greghaynestchaypo: just to test the rabbit parts I think23:12
howleyttchaypo: I want to try out rabbitmq cluster23:12
tchaypoah, very good.23:12
SpamapSaha23:13
greghaynesok, enough internal videos for the week... time to code23:13
SpamapShttp://vim.wikia.com/wiki/How_to_stop_auto_indenting23:13
SpamapSfixed it23:13
*** cwolferh_ has quit IRC23:14
*** derekh has joined #tripleo23:15
*** sseago has joined #tripleo23:17
lifelesshowleyt: so, what you could do is make a template that just configures rabbit on two nodes and build an image with just rabbit on it23:18
openstackgerritClint "SpamapS" Byrum proposed a change to openstack/tripleo-heat-templates: WIP: Really ugly first switch to software-config  https://review.openstack.org/8166623:18
*** weshay has quit IRC23:19
SpamapSstevebaker: ^^ there are a million things wrong.. but I have made "progress"23:19
stevebakerSpamapS: does anything actually work?23:21
Lotus907efihmmm, overcloud  CREATE_FAILED23:22
Lotus907efiovercloud-notCompute0-mdwo3l7idqo7  | ERROR23:22
Lotus907efibut overcloud-NovaCompute0-k55pidjqgfn5 | ACTIVE | RUNNING23:22
Lotus907efiand overcloud-NovaCompute1-nijjq5jmxx3o | ACTIVE | Running23:22
SpamapSstevebaker: no23:23
SpamapSstevebaker: need to fix merge.py a little more, or the templates.. not sure23:23
SpamapSoops, added a .tmp file23:24
SpamapSanyway..23:25
* SpamapS -> kids23:25
stevebakerSpamapS: looking now23:25
*** cwolferh_ has joined #tripleo23:29
* greghaynes tries to learn what software-config is about23:29
*** epim has quit IRC23:30
stevebakergreghaynes: any questions let me know23:30
greghaynesAny docs you know of? At the moment im just reading tests23:30
stevebakergreghaynes: I'm writing them now. There is this https://review.openstack.org/#/c/81407/ https://review.openstack.org/#/c/81629/ https://review.openstack.org/#/q/project:openstack/heat-templates+topic:software_config,n,z23:31
greghaynesaha!23:32
*** epim has joined #tripleo23:37
*** spzala has quit IRC23:38
*** spzala has joined #tripleo23:38
*** spzala has quit IRC23:39
tchaypoI wonder if HP has a corp account with Safari23:39
*** spzala has joined #tripleo23:40
*** epim has quit IRC23:42
howleytlifeless: sounds like a good plan also, might force me learn what is actually happening with heat and the yaml configs.23:43
howleytI'm going to sign off, folks, bedtime for me. Thanks for the help.23:45
lifelessnight23:46
*** howleyt has quit IRC23:47
*** epim has joined #tripleo23:51
*** rpodolyaka1 has joined #tripleo23:55
*** rpodolyaka1 has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!