Wednesday, 2018-11-07

kmalloclbragstad: i think we can just not set headers in KSM instead of setting them to None00:08
kmallocto be WSGI compliant00:08
openstackgerritMorgan Fainberg proposed openstack/keystonemiddleware master: [WIP] Correct auth_token headers to be WSGI compliant
openstackgerritMorgan Fainberg proposed openstack/keystonemiddleware master: [WIP] Correct auth_token headers to be WSGI compliant
openstackgerritMorgan Fainberg proposed openstack/keystonemiddleware master: [WIP] Correct auth_token headers to be WSGI compliant
*** irclogbot_1 has quit IRC00:38
*** erus has quit IRC01:00
wxy-xiyuangagehugo: for is it OK now? Since the related deprecation patch has been merged already. Or any other part need be improved?01:22
*** aojea has joined #openstack-keystone01:23
*** _01000101_ has joined #openstack-keystone01:25
_01000101_Good evening. Should keystone (master) should pass the py27 tests without modification? I'm getting test failures and want to make sure my env is OK before I proceed with a change.01:27
*** aojea has quit IRC01:27
wxy-xiyuan_01000101_: what error you hit? generally, it should be passed by `tox -e py27` command.01:28
_01000101_keystone.tests.unit.test_sql_upgrade.FullMigration.test_migration_024_add_created_expires_at_int_columns_password -----------------------------------------------------------------------------------------------------------------  Captured traceback: ~~~~~~~~~~~~~~~~~~~     Traceback (most recent call last):       File "keystone/tests/unit/", line 2371, in test_migration_024_add_created_expires_at_int_01:34
_01000101_Hmm, one sec, let me use pastebin01:34
_01000101_The errors mostly look the same / similar to the one posted01:35
*** Dinesh_Bhor has joined #openstack-keystone01:36
wxy-xiyuan_01000101_: is sqlite installed ?01:36
_01000101_I simply did `pip install -r test-requirements.txt` (like what I would do for other projects) and then right to tox. I'm installing sqlite3 now01:37
_01000101_wxy-xiyuan: I re-ran with sqlite3 installed but I got the same failures01:43
wxy-xiyuan_01000101_: emm, so only 18 tests case failed. are all errors the same?01:44
_01000101_That about sums it up. It's a huge amount of output between failures, but they mostly look like this -     oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) error in trigger federated_user_insert_trigger: no such table: main.migration_tmp [SQL: u'ALTER TABLE federated_user RENAME TO migration_tmp'] (Background on this error at:
*** markvoelker has quit IRC02:00
*** markvoelker has joined #openstack-keystone02:00
_01000101_It looks like someone had the same issue recently -
openstackDebian bug 909989 in src:keystone "keystone: FTBFS (failing tests)" [Serious,Fixed]02:01
*** markvoelker has quit IRC02:02
wxy-xiyuan_01000101_: it works well in my local env. I guess it's related to some libraries version problem.02:03
_01000101_I've submitted patches for Nova and CinderClient very recently without issue using the same steps for testing. I'm running latest OpenSUSE and latest packages from PyPi02:04
wxy-xiyuan_01000101_: So the package just skipped related tests. It is perhaps still a problem in Keystone. I'll trying to reproduce it locally. Feel free to register a bug in Launchpad. Thanks.
_01000101_Roger that, thanks02:18
_01000101_ filed02:40
openstackLaunchpad bug 1802035 in OpenStack Identity (keystone) "Master branch failing py27 tests (oslo_db.exception.DBNonExistentTable:)" [Undecided,New]02:40
*** felipemonteiro has joined #openstack-keystone02:51
*** _01000101_ has quit IRC02:58
openstackgerritVishakha Agarwal proposed openstack/keystone master: Add abstract method in trusts
*** felipemonteiro has quit IRC03:16
*** felipemonteiro has joined #openstack-keystone03:37
*** felipemonteiro has quit IRC04:15
*** Dinesh_Bhor has quit IRC04:15
*** jmlowe has quit IRC04:21
*** felipemonteiro has joined #openstack-keystone04:31
*** Dinesh_Bhor has joined #openstack-keystone04:34
*** sapd1_ has quit IRC04:35
*** sapd1 has joined #openstack-keystone04:36
*** felipemonteiro has quit IRC04:43
*** felipemonteiro has joined #openstack-keystone04:47
*** felipemonteiro has quit IRC04:53
openstackgerritVishakha Agarwal proposed openstack/keystone master: Fixing nits
*** jrist has quit IRC05:58
openstackgerritGhanshyam Mann proposed openstack/keystone master: DNM: testing system_scope
*** jrist has joined #openstack-keystone06:11
*** aojea has joined #openstack-keystone06:27
*** aojea has quit IRC06:31
*** felipemonteiro has joined #openstack-keystone06:50
*** andreaf has quit IRC06:53
*** andreaf has joined #openstack-keystone06:55
openstackgerritwangxiyuan proposed openstack/keystone master: Refresh health check doc
*** pcaruana has joined #openstack-keystone07:36
openstackgerritwangxiyuan proposed openstack/keystone master: Refresh admin doc
*** felipemonteiro has quit IRC08:06
*** BlackDex has quit IRC08:20
*** BlackDex has joined #openstack-keystone08:33
*** amoralej|off is now known as amoralej08:45
*** dims has quit IRC08:52
*** dims has joined #openstack-keystone08:53
*** dims has quit IRC08:58
*** dims has joined #openstack-keystone08:59
*** kukacz has quit IRC09:04
*** kukacz has joined #openstack-keystone09:05
*** Dinesh_Bhor has quit IRC09:43
openstackgerritMerged openstack/keystone master: Add py36 tox environment
openstackgerritMerged openstack/keystone-tempest-plugin master: Add python3 functional test job
openstackgerritMerged openstack/keystonemiddleware master: Stop supporting revocation list
*** ondrejme has joined #openstack-keystone10:27
ondrejmeHi, tempest IdentityV3UsersTest keep failing on "The request you have made requires authentication", and in debug logs i see 'X-Auth-Token': '<omitted>'. I think the problem is because http requests don't include this X-Auth-Token. How can i include it to the message please? I run on ocata.10:29
cmurphyondrejme: if you see x-auth-token in the keystone logs that means it's coming through in the request, the omitted is just to avoid leaking sensitive data in the logs10:31
ondrejmeOh, i see10:32
cmurphyondrejme: if you turn on insecure_debug in the keystone logs it should give you more information about why the authentication failed10:32
cmurphyis your tempest admin user configured with the correct credentials?10:32
ondrejmeIt should be, yes. I will check insecure_debug logs, thanks for now10:33
ondrejmeI asked about the token because logs also say:10:36
ondrejmeINFO keystone.middleware.auth [req-26d6f611-971c-4027-8001-2be61ea5bfa0 016aafa0ba9f4a50aacb3f9fe47226f8 f22b586f94e04669a6bc6035427a886c - default default] Invalid user token10:36
cmurphyondrejme: are you using uuid or fernet tokens?10:36
cmurphythat message could mean the fernet key repository is misconfigured10:36
cmurphyare you able to authenticate at all? or is it just tempest having the problem?10:37
ondrejmeIt's just tempest10:37
cmurphywell insecure_debug should make it clear what's wrong10:37
*** Dinesh_Bhor has joined #openstack-keystone10:37
ondrejmeok, ill check10:38
ondrejmeThis looks a bit better10:47
ondrejmeWARNING keystone.common.wsgi [req-7e8aed75-6f4b-4f31-8150-b2ac647ee5f9 - - - - -] Authorization failed. Invalid username or password (Disable insecure_debug mode to suppress these details.)10:48
cmurphyyeah so looks like the credentials in your tempest conf are wrong10:48
ondrejmebut in tempest.conf im sure that the "admin_username" and "admin_password" are correct10:48
ondrejmei deploy through kolla10:48
ondrejmeand i check password in passwords.yaml10:48
ondrejmeshould i provide different credentials to tempest?10:49
cmurphytry using the openstack client with those credentials to check if they really work?10:50
ondrejmei just checked, they work (tried a bad password too and it didn't)10:59
ondrejmeCan swift's proxy-server section [filter:authtoken] have anything to do with this?11:01
cmurphyI don't think so, it shouldn't interfere with direct requests to keystone11:01
cmurphydid you update tempest recently? maybe they changed how the auth params are set up11:02
ondrejmeThis is my tempest conf:
cmurphyi'm not a tempest expert11:04
cmurphyreading the docs i wonder if use_dynamic_credentials has anything to do with it11:04
ondrejmeHundreds of other tests pass with this setup though11:05
ondrejmewithout authorization problem11:05
cmurphyokay so it's not a problem with the admin credentials then, just the Users test11:06
cmurphyis it the test_user_update_own_password test that's failing?11:07
ondrejmealso 2 more11:07
cmurphywhich others?11:07
cmurphythe Authorization failed message in the logs is expected for the update password test
cmurphyso what's the actual failure from tempest?11:08
ondrejmetempest.lib.exceptions.Unauthorized: Unauthorized11:09
ondrejmeDetails: Invalid username or password (Disable insecure_debug mode to suppress these details.)11:09
cmurphycan you paste the whole test output?11:10
ondrejmesure, gimme a sec11:10
cmurphyunauthorized is expected at least at some point11:10
ondrejmein keystone logs i can also see:11:11
ondrejme2018-11-07 11:10:43.119 20 DEBUG keystone.middleware.auth [req-b37c545f-a8b4-4c8f-894e-2fd7b28dd898 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set11:11
cmurphythat's benign11:11
cmurphythat's normal11:11
ondrejmekk, ill fetch the tempest logs11:12
*** sapd1 has quit IRC11:12
*** sapd1 has joined #openstack-keystone11:12
ondrejmelog for the three tests11:15
*** aojea_ has joined #openstack-keystone11:18
cmurphydo you have password restrictions set in [security_compliance] in keystone.conf? or any logs about password restrictions in the logs?11:20
ondrejmenope, nothing like that in my keystone.conf11:22
*** aojea_ has quit IRC11:23
cmurphyit kind of seems like a timing issue to me11:24
cmurphytwo of the tests are failing on _restore_password which means they were able to successfully change the password once11:24
cmurphythe other is failing because it's expecting to have the user locked out but they aren't locked out yet11:24
cmurphyondrejme: what version of tempest are you using?11:26
cmurphybecause I think we fixed this
ondrejmei checked for these changes earlier when debugging, they are present in the code11:28
cmurphyi'm out of ideas then11:29
*** Dinesh_Bhor has quit IRC11:33
ondrejmeOk, here's a twist11:47
ondrejmewe actually use Master tempest11:47
ondrejmeon ocata11:47
cmurphyi don't think we test master tempest on ocata keystone but in any case those tests haven't changed much and the sleep still applies11:49
*** raildo has joined #openstack-keystone12:24
ondrejmeThe weird thing is that this only happens on identity v3 tests..12:38
*** amoralej is now known as amoralej|lunch13:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/keystone master: Remove redundant variables from context class
openstackgerritMerged openstack/ldappool master: Allow pool status to be printed as a table
openstackgerritMerged openstack/keystone-specs master: fix misspelling of configuration
openstackgerritMerged openstack/keystone-specs master: Update policy security roadmap
*** pcaruana has quit IRC13:45
*** jmlowe has joined #openstack-keystone13:47
*** pcaruana has joined #openstack-keystone14:00
*** aojea_ has joined #openstack-keystone14:04
*** amoralej|lunch is now known as amoralej14:25
*** pcaruana has quit IRC14:33
*** pcaruana has joined #openstack-keystone14:34
*** jistr is now known as jistr|call15:00
*** ondrejme has quit IRC15:03
*** aojea_ has quit IRC15:04
gagehugowxy-xiyuan: the ps is still deleting that function, it should be kept in if we are deprecating it now right?15:05
*** felipemonteiro has joined #openstack-keystone15:16
*** jistr|call is now known as jistr15:29
*** aojea_ has joined #openstack-keystone15:30
cmurphyknikolla: i have a band-aid for the federation tests if you want to have a look
knikollacmurphy: thanks! easy +2!15:34
*** aojea_ has quit IRC15:35
*** aojea_ has joined #openstack-keystone15:35
*** aojea_ has quit IRC15:35
cmurphyknikolla: kmalloc the deadline to select outreachy interns is on monday, can we sync up on picking people soonish?15:35
cmurphyi'm stressed about picking from so many good candidates :(15:36
*** aojea_ has joined #openstack-keystone15:36
knikollacmurphy: sure. do we want to schedule a bluejeans call?15:36
cmurphyknikolla: if you have a bluejeans account sure15:36
knikollai got locked out of it since i got a new phone and forgot to add it as a 2FA device :/ but i'm sure kmalloc does have one.15:38
knikollaotherwise hangouts or any other service should work.15:38
*** felipemonteiro has quit IRC15:39
*** david-lyle has quit IRC15:42
*** dklyle has joined #openstack-keystone15:47
*** aojea_ has quit IRC15:56
*** aojea_ has joined #openstack-keystone15:57
*** gyee has joined #openstack-keystone15:58
kmalloccmurphy: send the link to his redhat account. :)16:04
kmalloccmurphy: yeah I can schedule blue jeans in a second.16:04
*** aojea_ has quit IRC16:06
kmallocI am free anytime post coffee and have blue jeans ready.16:07
kmallocknikolla, cmurphy ^16:07
cmurphyi could join in about 20 minutes probably16:08
cmurphyor could do later tonight or tomorrow if you guys want time to review the applications16:09
knikollaeither way works for me. i'm free in about about 15 minutes until the hour. then will be free after 2 hours16:11
*** imacdonn has quit IRC16:17
*** imacdonn has joined #openstack-keystone16:17
kmallocLet's do later16:31
kmallocOr we could do now.16:32
kmallocHold on let me get to the computer16:32
cmurphyi'm here16:32
cmurphywe can do a quick session now and do another tomorrow if we need to16:33
kmalloccmurphy, knikolla:
kmallocnow works for me.16:34
*** pcaruana has quit IRC16:38
kmallocknikolla: lets plan for when you're done in ~2hrs (2pm Eastern ish?)16:45
kmalloci can't do timezone math16:45
knikollakmalloc: sure. 1.30pm eastern works for me.16:46
kmalloccool, cmurphy ^16:47
kmalloccmurphy: also i'm now approved as a mentor16:47
kmallocso. yay!16:47
samueldmqeasy approval \o/16:47
kmallocOH it's You! :)16:47
kmallochi samueldmq :)16:47
samueldmqheh :-) hi kmalloc and cmurphy16:48
kmallocsamueldmq: submitted for the other one too :P16:48
samueldmqkmalloc: awesome, approved!16:49
cmurphyhaha kmalloc ignore my google doc invite then ;)16:49
kmalloci assume samueldmq is _not_ going to be in berlin?16:49
kmallocbecause ... it's always good to see him in person.16:50
samueldmqkmalloc: unfortunately your assumption is correct16:50
samueldmqkmalloc: likewise thanks16:50
samueldmqkmalloc: I'm finishing to write my masters dissertation on intra-platform interop in openstack16:50
samueldmqmeaning I'll have plenty of things to prsent next time16:51
kmallocsamueldmq: nice.16:51
* kmalloc is planning on going back to school (part time) for an advanced degree soon as well.16:52
kmallocreminds me i need to sign up for the "entrance" exam.16:52
samueldmqkmalloc: ++ I really enjoy studying, it's hard to balance things, but I like to suffer perhaps16:53
lbragstadsamueldmq kmalloc you two must have figured out time travel or something...16:57
samueldmqlbragstad: o/16:59
samueldmqlbragstad: oh sure, that's the same as dark circles under eyes16:59
lbragstadi tried washing the dark circles off.. it didn't work17:00
samueldmqlbragstad: I heard having more babies helps out17:01
kmalloclbragstad: nah, i just don't sleep17:01
lbragstadsamueldmq does it? that math doesn't work17:01
*** irclogbot_1 has joined #openstack-keystone17:02
kmalloclbragstad: ask steve17:09
kmalloci'm sure he can tell you17:09
openstackgerritMerged openstack/keystone master: Switch devstack plugin to
*** aojea_ has joined #openstack-keystone18:26
knikollakmalloc: cmurphy: ready when you are.18:31
kmallocthnx for the ping18:31
kmallocsamueldmq: ping need you to approve knikolla for the unit tests one too18:41
kmallocsamueldmq: if ytou don18:41
kmalloc't mind18:41
knikollasamueldmq: just sent the application18:43
samueldmqkmalloc: knikolla: my pleasure, done! thanks for doing so18:45
*** aojea_ has quit IRC18:57
kmallocknikolla: i have a strange idea on federation testing...18:57
knikollakmalloc: go on18:57
kmallocknikolla: I'll stand up a dev server with vexxhost today we can install shib on for testing at least that way we control it18:57
kmallocand we can run an OIDC provider on it too18:57
kmalloci need to standup ipsilon for some other things (testing) in infra anyway18:58
kmallocwe can keep things non-voting but we'll at least fully own end-to-end the provider we're testing against18:58
knikollasure. that sounds fine.18:58
kmallocrahter than rely on "broken" or "well who knows" public things like testshib/saml.id18:58
knikollafor now.18:58
cmurphykmalloc: did you see that i got the tests green?18:58
kmalloccmurphy: yes i did!18:59
kmalloccmurphy: it was why i was thinking we should at least own the test server18:59
kmalloceven if it isn't stood up in gate dynamically18:59
cmurphyif we can set it up ourselves then we can stand it up in the gate18:59
kmallocthat is the plan18:59
cmurphybut i don't want to have to be the one to set up a shibboleth idp18:59
kmallocstand it up, and then replicate to gate19:00
cmurphyor maintain one19:00
kmallocso we use this as a "what are the steps" bits to get it right19:00
*** amoralej is now known as amoralej|off19:00
kmallocand we maintain a change to point to it so we can confirm our tests work like they should19:00
kmalloc(doesn't have to merge)19:00
kmallocbut i have to standup ipsilon for infra reasons.19:01
knikollakmalloc: please document extensively the standing up process19:01
kmallocso, might as well re-use that19:01
kmallocknikolla: hehehe19:01
kmallocknikolla: it will be done via ansible so i can repeat it19:01
kmalloci have zero desire to hand-build anything19:01
cmurphyor containerize it so we don't have to understand it :P19:01
knikollagotta love infra as code19:01
kmalloccmurphy: well, ansible + docker.19:02
kmallocbecause that plays into infra / zuul nicely19:02
kmallocbut first steps: ansible and make it repeatable19:02
kmallocthe biggest question is the TLS/PKI setup19:02
kmalloca self-signed easy-ca should be ok, right? for testing?19:03
cmurphythe biggest question is java--19:03
knikollaha, ha, h19:03
kmalloci was actually looking at ipsilon (python) and seeing if it can run in isolation19:03
kmallocrather than shib.19:04
kmallocas long as we are running saml2, OIDC, i'm happy19:04
cmurphyoh okay then19:04
knikollajava is a type of coffee beans, java is a type of coffee beans, java is a type of coffee beans...19:04
kmalloc`It is by caffeine alone I set my mind in motion. It is by the beans of Java that thoughts acquire speed, the hands acquire shakes, the shakes become a warning. It is by caffeine alone I set my mind in motion.`19:05
* kmalloc resists the urge to change the channel topic to that ^19:05
*** kmalloc is now known as needscoffee19:05
needscoffeeBRB. my new nick says it all.19:06
* knikolla hands needscoffee a cup of coffee19:06
needscoffeethe real question...19:06
needscoffeedo i drink ....19:07
cmurphyobvs the seasonal blend19:08
needscoffeeit's SO good19:09
* needscoffee wonders if there will be an issue bringing quality coffee with me to berlin in my bags.19:09
needscoffeeprobably an issue with customs.19:10
needscoffeeor some such19:10
needscoffeewith how much I drink that is :P19:10
needscoffeeoh, nice i should be ok as long as i bring less than 10kg19:11
needscoffeeyeah i am not drinking 10kg :P19:11
* needscoffee goes and drinks coffee... then writes code.19:12
needscoffeecmurphy going to post a change for oslo.cache to implement pymemcache (will also go upstream long term) and will replace both pool and python-memcache19:13
needscoffeecmurphy: i'll want your eyes on it because... there are very few people in openstack that I know that understand the insanity that is memoization caching19:14
cmurphyneedscoffee: i feel that you've grossly overestimated my understanding of caching19:15
needscoffeecmurphy: hhahaah nope.19:17
needscoffeeyou work on keystone, you have a strong grasp of what memoization really means and how it works.19:17
*** aojea_ has joined #openstack-keystone19:49
openstackgerritGage Hugo proposed openstack/keystone master: Region update extra support
*** nwilburn has quit IRC20:17
openstackgerritMerged openstack/keystone master: Fixing nits
*** needscoffee is now known as kmalloc20:19
kmalloccmurphy: this new memcache client setup is *so much* better20:20
kmallocand will make our caching superior. pooling is built-in20:20
kmallocwe can deprecate all the ick20:20
*** aojea_ has quit IRC20:21
*** aojea_ has joined #openstack-keystone20:21
*** aojea_ has quit IRC20:43
*** aojea_ has joined #openstack-keystone20:46
*** pas-ha has quit IRC20:47
*** hogepodge has quit IRC20:47
*** pas-ha has joined #openstack-keystone20:48
*** mnaser has quit IRC20:48
*** lamt has quit IRC20:48
*** hogepodge has joined #openstack-keystone20:48
*** mnaser has joined #openstack-keystone20:49
*** andreaf has quit IRC20:51
*** andreaf has joined #openstack-keystone20:52
*** aojea_ has quit IRC20:56
*** aojea_ has joined #openstack-keystone20:58
openstackgerritGage Hugo proposed openstack/keystone master: DNM - zuul testing
*** raildo has quit IRC21:02
*** aojea_ has quit IRC21:05
*** aojea_ has joined #openstack-keystone21:06
*** mchlumsky has quit IRC21:33
openstackgerritGage Hugo proposed openstack/keystone master: DNM - zuul testing
openstackgerritLance Bragstad proposed openstack/keystone master: Add scope documentation for service developers
openstackgerritGage Hugo proposed openstack/keystone master: Change __all__ list to tuple
gagehugothere's a lot of UserWarning: Invalid uuid: <<keystone.domain.root>>. To ensure interoperability, identifiers should be a valid uuid. in the unit test logs :(22:52
openstackgerritGage Hugo proposed openstack/keystone master: Region update extra support
gagehugolbragstad: done22:58
gagehugo"pls no use extra"22:58
kmallocgagehugo: yes. we should be exempting the root domain from CADF unless we extract it and make it a 1st order thing, which case, we probably want to do soemthing intellegent like sha515 <<keystone.domain.root>>>[:32] as the ID (migration)22:58
kmallocand wire up a minor bit of compat code to handle the case where id=<<keystone.domain.root>>22:59
* kmalloc is strongly of the opinion we should unhide the root domain and make it the global top-level (any roles granted on it are explicitly inherited only, no way to scope to the root domain directly)22:59
gagehugokmalloc: exempting should be pretty easy23:00
kmallocgagehugo: right. we should exempt it in keystone.23:00
kmallocand never emit cadf if we're on that domain23:00
gagehugooh I see23:00
kmallocand like i said, we should unhide that domain in general and make ti the place where grants go that we want inherited down the whole tree23:01
kmalloclbragstad, gagehugo: I'd rather reverse the contract breaking decision and drop the extra column for regions23:01
kmallocsince we're in a API contract break in either case23:01
kmalloca) we allow at creation but not on update23:02
kmallocb) we don't allow on creation or update23:02
kmallocmy opinion is we just drop the "extra values" on the floor.23:02
* kmalloc will comment on that review.23:02
kmallocgagehugo: commented on the review (cc lbragstad) with a -1 and a note that other cores should override my -1 if it should land as is.23:04
gagehugoI'm fine with either choice23:04
kmalloci am not advocating eliminating the "additional_attributes" part of the json_schema because that would break creation requests23:04
kmallocbut i really would rather just drop extras to the floor23:04
kmallocif we need the concept of extras we can add it back in, but in a well-defined location, e.g. [Resource-Object]->VendorData23:05
gagehugoI was just picking up work to close out bugs :p23:05
kmallocanyway, like i said, i vote for dropping the additional cruft on the floor on create and contracting out that column23:06
gagehugosounds like a good Berlin topic too :)23:06
kmallocif we could drop the extras columns everywhere, i would.23:07
kmallocthat design choice was one of the worst things we've ever had in keystone23:07
kmallocit has also resulted in things like passwords being stored in plain text (historically)23:07
*** aojea_ has quit IRC23:24
*** aojea_ has joined #openstack-keystone23:27
*** aojea_ has quit IRC23:32

Generated by 2.15.3 by Marius Gedminas - find it at!