Got stuck with large amounts of files

The place for threads about version 1.x of TimoSoft ExplorerTreeView.
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Got stuck with large amounts of files

Post by WoF »

Hello Tiku,

it's me again (by now still satisfied user of V1.10.0) and I'm back with a rather annoying problem.
My app has to deal with folders containing large amounts of files.
Assume a structure on disk where a folder named "Film1" has 10 subfolders "Scene1" to "Scene10" each containing about 20000 to 25000 images, often rather large files. When you first select the parent folder "Film1" you are f.. er, doomed. You are stuck for a couple of minutes until all 10 subfolders (which contain the large amounts of files) are displayed.
It seems as if the treeview looks ahead and scans the contents of the folders.
Is there a flag or option or property in ExplorerTreeView which disables this looking-ahead? I think, it is really not necessary to scan through all the files. Or is there any chance you can integrate such a property in the next version?
That would be a great help 'cause it's sooo time consuming otherwise...
Looking forward to a reply.
Best regards
Wolfgang
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

Hi,

when inserting items, ExplorerTreeView determines whether they should be expandable. Depending on the settings, it has to enumerate the sub-items of the inserted item until it finds one, that actually would be displayed if the inserted item gets expanded.
This check for sub-items can be time-consuming, but I doubt a bit, that it is the reason here, because the check is done only when you expand an item for the first time and not if you simply select it.
Are you displaying info tips? In other words: if you hover over a folder, does there appear a tooltip displaying the folder's content and size? If yes, turn it off and see whether performance becomes better.
I'd also like to see your settings. Please store them to a file (using the "Save" button on the property pages or the SaveSettingsToFile() method) and attach it here.

TiKu
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Hi Tiku,
thanks for your fast reply 'cause I get literally stuck.
I don't display tooltips.
You seem to be right about the enumeration determining whether a folder should be expandable (getting the "+").
As I said, when I expand a parent which contains say 10 folders each of which holds 25000 files it takes considerable time until all 10 folders are shown. This comes one after the other whith remarkable harddisk rattling, when it looks for subfolders therein. At this time I have not even selected one of those folders. I only opened the parent.
Also remarkable is, when I go elsewhere in the tree and come back to the problem-folder, it goes rather fast and no harddisk operation is audible. I think the enumerated item names are still in some cache memory of Windows, so the process is considerably faster. But it is not guaranteed to remain so. After a while working you will have the same time consuming behaviour as before.
I attached my settings. Hope you can figure out something.
Best regards
Wolfgang
Attachments
WoFsExplorerTree.zip
My settings
(517 Bytes) Downloaded 608 times
Last edited by WoF on 08 Aug 2006, 12:38, edited 1 time in total.
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

I can't see the attachment.;)
Anyway, the sub-items check already is very optimized, so I think I'll implement a new property "DontCheckFolderExpandibility". The result will be that all folders get a "+". If a folder doesn't contain visible sub-items, this "+" will simply vanish if you click on it.
Would this be okay?
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Sorry, I was too dumb to note that I'd have to zip the file before attaching it. :roll:

Yes this would certainly do the trick. That would be the ultimate solution for me. Really phantastic.

What would that mean for my app and the already installed programs?
Will this result in a new control to rplace the old one on my form so that I have to unregistrate the old one and registrate the new one?
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

WoF wrote:Sorry, I was too dumb to note that I'd have to zip the file before attaching it. :roll:
No problem.
WoF wrote:Yes this would certainly do the trick. That would be the ultimate solution for me. Really phantastic.
Okay, see the attached build. It will probably be released as version 1.12.0 within the next 7 days.
WoF wrote:What would that mean for my app and the already installed programs?
Will this result in a new control to rplace the old one on my form so that I have to unregistrate the old one and registrate the new one?
Build 800 is backward compatible to version 1.11.1, which is backward compatible to 1.11.0. Unfortunately there was a compatibility break somewhere between 1.10.0 and 1.11.0. This means you'll have to recompile your app if you want to use 1.12.0. Sorry. On the other hand you need a recompile anyway, if you want to use the new feature (it defaults to "False", so old programs keep their behavior).
However, there's no need to remove the old control from your form and replace it with the new one. Simply register the new one, change "DontCheckFolderExpandibility" to "True" and recompile. If this doesn't work, let me know. VB6 sometimes messes up registry entries.
Attachments
extvw800.zip
Version 1.12.0 (Build 800) of ExplorerTreeView
(235.4 KiB) Downloaded 613 times
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Wow, you're the Master of Trees. In my View, anyway. ;)

Thanks, Tiku, everything worked out alright. I simply replaced the .ocx and registrated again and the new property was available after restarting the IDE. Perfect. Phantastic. Now everything is as fast as could be. You can not imagine how annoying it was to wait for 10 folders to be scanned.
I'm really grateful for your your fast response and immediate help. Thank you very much.

BTW.: is there a means to inhibit the trashcan from appearing in the tree?

Best regards
Wolfgang
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

WoF wrote:Wow, you're the Master of Trees. In my View, anyway. ;)
Thanks. :)
WoF wrote:Perfect. Phantastic. Now everything is as fast as could be. You can not imagine how annoying it was to wait for 10 folders to be scanned.
It would be interesting how fast version 1.12.0 is without this new feature. Last christmas I rewrote big parts of the control and it should be much faster than version 1.10. However, I never did any real benchmarks. Your folder structure would be a great test. So maybe you could do a simple performance comparison between 1.10.0 and 1.12.0 (with "DontCheckFolderExpandibility" = "False")?
WoF wrote:BTW.: is there a means to inhibit the trashcan from appearing in the tree?
Some kind of. Make the "FireBeforeInsertItem" property include "fbiiForNonFSFolders". Then the "BeforeInsertItem" event should be raised for all virtual folders. All you have to do now is compare the pIDL you get in this event with the trashcan's pIDL and set "Cancel" to "True" if they match.
To retrieve the trashcan's pIDL, use either the "SHGetSpecialFolderLocation" or the "SHGetFolderLocation" API function with CSIDL_BITBUCKET. The first one works on Windows 95/NT 4.0 and newer, the latter one needs version 5.0 of shell32.dll. To compare the pIDLs, use the "ILIsEqual" API, which is defined as follows:

Code: Select all

Private Declare Function ILIsEqual Lib "shell32.dll" Alias "#21" (ByVal pIDL1 As Long, ByVal pIDL2 As Long) As Long
It returns a non-zero value if the pIDLs are equal. If you don't need the trashcan's pIDL anymore, free it using the "SHFree" API. You may also use ExplorerTreeView's "FreepIDLMem" method.

TiKu
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Ok. I play a little with the pIDL functions to avoid the trashcan.
Why do I have to free the memory? Is this not only some pointer to the trashcans pIDL or is this pIDL specially built for the caller of 'SHGetFolderLocation'?

I did compare speeds of both versions. I'm afraid I can't find any noticable difference from the feel of it, so I didn't bother to measure the time. The enumeration process when scanning a folder seems to take about the same time as in 1.10.0 maybe only a leeettle faster.
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

Okay, I'm somewhat disappointed by the results of your performance test. I expected the difference to be noticeable.

Concerning the pIDL: A pIDL is a pointer to an ItemIDList structure, which is an array of SHITEMID structures. This array holds information about the shell item it represents, e. g. the name. An ItemIDList is like a path. The Desktop is the shell's root, so the trashcan's ItemIDList contains 1 SHITEMID. The folder C:\Windows is represented by 3 SHITEMIDs: My Computer, C and Windows. You understand?
An ItemIDList structure isn't global. Each time you call SHGetFolderLocation, a new memory block is allocated and filled with the information about the shell item. So, yes, the memory, that the pIDL points to, is specially built for the caller of SHGetFolderLocation.
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Sorry to disappoint you. If there were any noticable difference I would have benchmarked it for you, but there wasn't.

Blocking out the trashcan worked perfectly, like you explained.
I get the CSIDL_BITBUCKET pIDL during Form_Load() and compare it in the BeforeInsertItem to the current pIDL. I free it at Form_Unload().
I like your concept to allow the BII event to be selected for non-FS items only. Good thinking there.

How could I make a pIDL visible for me, who is used to strings and likes to see what there is?
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

WoF wrote:How could I make a pIDL visible for me, who is used to strings and likes to see what there is?
Best thing is a debugger like Visual Studio, which offers a hex view of the memory.
You shouldn't take any data directly from the pIDL, because this data is private to the shell namespace that manages the item that the pIDL belongs to. There's no unique layout for pIDLs. Each of them can contain different kind of information and use a different format. There isn't even any real guarantee, that e. g. the trashcan's pIDL uses the same format on Windows XP SP2 as it does on Windows XP SP1.
The only thing that all pIDLs have in common, is that they are arrays of SHITEMID structures and SHITEMID is defined like this (Pseudocode):

Code: Select all

Private Type SHITEMID
  ' size of the whole SHITEMID structure (including cbSize)
  ' NOTE: cbSize actually is an unsigned short, but VB6's Integer is a signed short
  cbSize As Integer
  data(0 To cbSize - 3) As Byte
End Type
If you want to work with pIDLs, use the Shell API. Some keywords that should make you get started: IShellFolder, IShellFolder2, IContextMenu, IContextMenu2, IContextMenu3, SHGetDesktopFolder, IL*. There's also a large article about shell interfaces and pIDLs on MSDN, but I can't find it right now.
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Hm. That may sound stupid to you, but how do I enter the debugger willingly at the position where I have a valid pIDL to look at?
I only ever got to the debugger whena fatal error occured... :?
User avatar
TiKu
Administrator
Administrator
Posts: 832
Joined: 28 Sep 2004, 21:10
Location: München
Contact:

Post by TiKu »

You can attach the Visual Studio debugger to any process. In Visual Studio 2005 all you have to do is open a project (the "Debug" menu won't be displayed if no project is opened) and then choose "Attach to process" from the "Debug" menu. Once you've done so, you can choose "Debug"/"Windows"/"Memory"/"Memory X" to observe the debugged process' address space. You may have to pause the process with the debugger to actually see the address space, I'm not sure.
As I said, a pIDL is a pointer. So if the pIDL is &H12345678, go to address 0x12345678 and you should see the pIDL's content. I've never done it myself, but it should work this way.
Crunching for Fab36_Folding-Division at Folding@Home. Join Fab36/Fab30! - Folding@Home and BOINC
Boycott DRM! Boycott HDCP!
WoF
Lt. Commander
Posts: 43
Joined: 04 Aug 2005, 13:18

Post by WoF »

Sorry. I've got VS 6.0 and cannot find "Attach to process" in the Debug menu. Which Visual Studio Program are you talking about? The VB IDE?
Post Reply