An open community 
of Macintosh users,
for Macintosh users.

FineTunedMac Dashboard widget now available! Download Here

Previous Thread
Next Thread
Print Thread
Understanding Spotlight...
#10933 07/12/10 07:26 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
To the best of my knowledge I've disabled Spotlight with

Code:
sudo mdutil -a -i off

and

sudo rm -fr /.Spotlight-V100

but a search term entered in its icon's drop-down still yields file-name, not content, results.

After doing some research, I'm guessing that even though Spotlight has no index to search it is still searching metadata.

Is that correct (on a rudimentary level, anyhow)?

Thanks.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Understanding Spotlight...
artie505 #10935 07/12/10 10:16 AM
Joined: Aug 2009
Likes: 3
Moderator
Offline
Moderator

Joined: Aug 2009
Likes: 3

Quote:
To the best of my knowledge I've disabled Spotlight...

You can confirm this with

mdutil -s /

(replace / with the path to the volume if you're not checking the startup volume).

I don't know how/where the line is drawn between filenames and filesystem metadata, but you could try

mdls path_to_file

on one of the hits you get when using Spotlight; that'll tell you what, if any, metadata are "associated with" the file in question, which might shed some light on the issue.



dkmarsh—member, FineTunedMac Co-op Board of Directors
Re: Understanding Spotlight...
dkmarsh #10936 07/12/10 10:59 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
I'd already gotten the same response from
Code:
sudo mdutil -E /
as I got from
Code:
mdutil -s /
namely
Code:
Artie-s-Computer-4:~ artie$ sudo mdutil -E /
/:
	Indexing disabled.
Artie-s-Computer-4:~ artie$ mdutil -s /
/:
	Indexing disabled.
Artie-s-Computer-4:~ artie$ 
which is what confused me in the first place (Sorry for not having mentioned that.), so I guess
Code:
Artie-s-Computer-4:~ artie$ mdls /Volumes/HD2/Clutter/Me/Creative/Birds/\?\ -\ J/I’M\ BLONDE.rtfd 
kMDItemFSContentChangeDate = 2010-05-23 19:40:00 -0400
kMDItemFSCreationDate      = 2008-04-21 20:59:37 -0400
kMDItemFSCreatorCode       = ""
kMDItemFSFinderFlags       = 0
kMDItemFSHasCustomIcon     = 0
kMDItemFSInvisible         = 0
kMDItemFSIsExtensionHidden = 0
kMDItemFSIsStationery      = 0
kMDItemFSLabel             = 0
kMDItemFSName              = "I’M BLONDE.rtfd"
kMDItemFSNodeCount         = 1
kMDItemFSOwnerGroupID      = 80
kMDItemFSOwnerUserID       = 501
kMDItemFSSize              = 0
kMDItemFSTypeCode          = ""
Artie-s-Computer-4:~ artie$ 
is the definitive answer... All metadata.

Many thanks. smile


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Understanding Spotlight...
artie505 #10937 07/12/10 11:31 AM
Joined: Aug 2009
Likes: 3
Moderator
Offline
Moderator

Joined: Aug 2009
Likes: 3

It's significant that every attribute listed is a File System attribute (e.g. starts with kMDItemFS). Note that these comprise a small portion of the list of available attributes you can view via

mdimport -A

(or, much more pleasingly formatted, at Spotlight Metadata Attributes).

That suggests to this neophyte that disabling Spotlight doesn't disable the indexing of filesystem metadata.



dkmarsh—member, FineTunedMac Co-op Board of Directors
Re: Understanding Spotlight...
artie505 #10944 07/13/10 04:14 AM
Joined: Aug 2009
Likes: 1
Offline

Joined: Aug 2009
Likes: 1
When you search by file name, Spotlight just does a search of the computer's directory--it doesn't even look at the index. So blocking the formation of an index won't disable a search by name.


Photo gallery, all about me, and more: www.xeromag.com/franklin.html
Re: Understanding Spotlight...
tacit #10947 07/13/10 10:13 AM
Joined: Aug 2009
Likes: 3
Moderator
Offline
Moderator

Joined: Aug 2009
Likes: 3

Not to pick nits, but what does "search by file name" mean in this context? artie is doing a regular Spotlight search ("I've disabled Spotlight...but a search term entered in its icon's drop-down still yields file-name, not content, results"), and the results are only names.

Do you mean that every Spotlight search undertaken when indexing is enabled consists of both a lookup against previously-indexed metadata and a real-time search of filenames? And that when indexing is disabled, Spotlight only performs the latter? Or is it that Spotlight still performs both searches but returns no results from the former since there's no index?

I ask because I'm curious what happens in cases in which indexing has been disabled but the index has not been erased. Searching in such cases should return metadata as well as filenames, no? So for your assertion to be correct, Spotlight has to follow this logic: "search the directory for filename matches. If indexing is enabled, search the database for metadata matches. If indexing is disabled, search the database for metadata matches, unless the database is empty."

If the database is empty, seems more straightforward simply to search it and return no results.



dkmarsh—member, FineTunedMac Co-op Board of Directors
Re: Understanding Spotlight...
dkmarsh #10963 07/14/10 06:45 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
I can't begin to answer your question, but Spotlight Overview: How Does Spotlight Work?, my starting point for this thread, may offer some insight. (I note that the doc was "Last updated: 2007-05-27.")


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Understanding Spotlight...
dkmarsh #10993 07/16/10 12:06 PM
Joined: Aug 2009
Likes: 1
Offline

Joined: Aug 2009
Likes: 1
That's actually a really interesting question.

To my knowledge, Spotlight will always search the index if it exists even if indexing is disabled, if you do a search by content. I'm not 100% sure this is the case, but I have seen Spotlight use an index that hasn't been updated in a long time.

A search by file name always searches the disk directory regardless of whether or not indexing is enabled and regardless of whether or not an index database exists; disabling indexing and deleting the index doesn't prevent a search by file name.


Photo gallery, all about me, and more: www.xeromag.com/franklin.html
Re: Understanding Spotlight...
artie505 #11049 07/20/10 08:52 PM
Joined: Aug 2009
Offline

Joined: Aug 2009
Here is an Apple developer note updated in 2006:

http://developer.apple.com/macosx/spotlight.html

I have not been able to find out if the number of unique words indexed by Spotlight has been increased since Spotlight was introduced. The limitation might be important to scholars using multiple languages.


MicroMat Inc
Makers of TechTool
Re: Understanding Spotlight...
dkmarsh #11058 07/21/10 07:21 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
Am I misunderstanding something here?

Quoting the Apple doc to which I linked in post #10963...

Quote:
Spotlight provides fast desktop searching by extracting metadata in the background and storing the indexed metadata for future searches. When a query is made, the indexed metadata is searched for matching files. (Emphasis added)

Your question to tacit, though, presumes that the indexed metadata is what we call the Spotlight index...

Quote:
Do you mean that every Spotlight search undertaken when indexing is enabled consists of both a lookup against previously-indexed metadata and a real-time search of filenames? And that when indexing is disabled, Spotlight only performs the latter? Or is it that Spotlight still performs both searches but returns no results from the former since there's no index? (Emphasis added)

As I read, and imply from, the Apple doc, Spotlight uses the indexed metadata for a file search and the Spotlight index for a content search, which would explain why my disabled, theoretically index-less Spotlight behaves as described.

And this quote from MM T3's linked doc supports that...

Quote:
The technologies that power Spotlight are:

A database consisting of a high-performance meta-data store and content index that is fully integrated into the file system. (Emphasis added)

[....]

That suggests that Spotlight searches its "all-inclusive" database as respects any search term (within the search parameters, of course).

And addressing tacit's post that elicited yours...

Quote:
When you search by file name, Spotlight just does a search of the computer's directory--it doesn't even look at the index. So blocking the formation of an index won't disable a search by name.

The parameterless search I described was for a word I knew was in both a particular file's name and contents, and it seems that Spotlight found my word-as-file-name in the metadata portion of its database but could not find my word-as-content because of the lack of a content portion of the database.

Last edited by artie505; 07/21/10 07:40 AM. Reason: Cleanup

The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Understanding Spotlight...
artie505 #11082 07/22/10 01:57 PM
Joined: Aug 2009
Likes: 16
Moderator
Offline
Moderator

Joined: Aug 2009
Likes: 16
Originally Posted By: artie505
Am I misunderstanding something here?

Quote:
Spotlight provides fast desktop searching by extracting metadata in the background and storing the indexed metadata for future searches. When a query is made, the indexed metadata is searched for matching files. (Emphasis added)

Your question to tacit, though, presumes that the indexed metadata is what we call the Spotlight index...

Yes, but don't get too wrapped around the axle by the term "Spotlight Index". The Spotlight index is not necessarily a separate or stand alone entity. Spotlight can and does take advantage of existing volume structures to either be its index per. se. or to contain elements such as the Content Index that are uniquely created by Spotlight for its own use.

Quote:
Quote:
Do you mean that every Spotlight search undertaken when indexing is enabled consists of both a lookup against previously-indexed metadata and a real-time search of filenames? And that when indexing is disabled, Spotlight only performs the latter? Or is it that Spotlight still performs both searches but returns no results from the former since there's no index? (Emphasis added)

As I read, and imply from, the Apple doc, Spotlight uses the indexed metadata for a file search and the Spotlight index for a content search, which would explain why my disabled, theoretically index-less Spotlight behaves as described.

Not all Apple documents are well written and even those that are may unintentionally mislead. Consider this, regardless of what search argument or what type of search argument you use the object found must always be a file. Even folders, packages, and apps are special cases of the object "file" in OS X/Unix. In this case the meta data may be any number of things such as keywords, Spotlight comments, type/creator codes, etc.

Quote:
Quote:
The technologies that power Spotlight are:

A database consisting of a high-performance meta-data store and content index that is fully integrated into the file system. (Emphasis added)

[....]

That suggests that Spotlight searches its "all-inclusive" database as respects any search term (within the search parameters, of course).

Here again there are a huge number of characteristics that Spotlight can use in its search. Among those are the content index which is generated by Spotlight (or at least the underlying Unix functions) and the infrequently used keyword or "Spotlight comment" index. Sherlock in earlier versions of OS X stored all of this in separate index files, which for one reason or another, proved to be rather fragile while Spotlight uses space in the volume structure for much or all of this. A handy part of putting all this in the volume/directory is the ability to take advantage of an already highly optimized set of search functions in the OS.

Quote:
Quote:
When you search by file name, Spotlight just does a search of the computer's directory--it doesn't even look at the index. So blocking the formation of an index won't disable a search by name.

The parameterless search I described was for a word I knew was in both a particular file's name and contents, and it seems that Spotlight found my word-as-file-name in the metadata portion of its database but could not find my word-as-content because of the lack of a content portion of the database.

You better hope your word-as-file-name search worked. Since Spotlight uses the directory structure of the volume for file name searches, it that had not worked it would be because the volume structure of the disk had been trashed.


If we knew what it was we were doing, it wouldn't be called research, would it?

— Albert Einstein
Re: Understanding Spotlight...
artie505 #11087 07/22/10 03:50 PM
Joined: Aug 2009
Likes: 3
Moderator
Offline
Moderator

Joined: Aug 2009
Likes: 3

As Joe noted, every result returned by any Spotlight search is a file. From the document linked to by MicroMat Tech3 above:

Quote:
The Spotlight Store is a file system-level database that holds all of the meta-data attributes about the files, as well as an index of their contents, on a file system. As each file is created, copied, updated, or deleted, Spotlight will ensure that both the content index and the meta-data store entries for that file are updated.

Nowhere in this document (nor in any other Spotlight resource I've been able to find) is it stated that filenames are the exclusive province of the content index; indeed, the fact that there exists a collection of kMDItemFS... metadata attributes refutes such a conclusion.

Also from that document:

Quote:
Some kinds of meta-data, such as file modification dates, ownership, and access permissions are kept external to the file by the file system and have been accessible via a variety of mechanisms.

If kMDItemFS... metadata attributes are in fact what is being referred to here, then they clearly won't be found in a Content Index; they can't be part of a file's content if they're "kept external to the file!"



dkmarsh—member, FineTunedMac Co-op Board of Directors
Re: Understanding Spotlight...
dkmarsh #11187 07/29/10 06:40 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
> Nowhere in this document (nor in any other Spotlight resource I've been able to find) is it stated that filenames are the exclusive province of the content index; indeed, the fact that there exists a collection of kMDItemFS... metadata attributes refutes such a conclusion.

I never even so much as suggested that filenames are the exclusive province of the content index; I questioned the fact that you seemed to have done so:

Quote:
Your question to tacit, though, presumes that the indexed metadata is what we call the Spotlight index...

Quote:
Do you mean that every Spotlight search undertaken when indexing is enabled consists of both a lookup against previously-indexed metadata and a real-time search of filenames? And that when indexing is disabled, Spotlight only performs the latter? Or is it that Spotlight still performs both searches but returns no results from the former since there's no index? (Emphasis added)

As I read, and imply from, the Apple doc, Spotlight uses the indexed metadata for a file search and the Spotlight index for a content search, which would explain why my disabled, theoretically index-less Spotlight behaves as described.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire

Moderated by  alternaut, dianne, dkmarsh 

Link Copied to Clipboard
Powered by UBB.threads™ PHP Forum Software 7.7.4
(Release build 20200307)
Responsive Width:

PHP: 7.4.33 Page Time: 0.023s Queries: 40 (0.018s) Memory: 0.6509 MB (Peak: 0.7649 MB) Data Comp: Zlib Server Time: 2024-03-28 11:25:08 UTC
Valid HTML 5 and Valid CSS