Possible new FileListToArray method for AutoIt

[The Freezer]

Got the following from Onepiece via BP -- with some slight modifications from me. It does seem faster, so later I'll do some comparative timings with SetupS between the two methods. If it's significantly faster -- the claims are as high as 20% -- then I'll adapt it to SetupS (v8 & v9).

Code: Select all

#include <Constants.au3>
#include <File.au3>
#include <Array.au3>
#include <Date.au3>

Local $StopWatch

;Test original _FileListToArray
Timer($StopWatch) ; start timer
Local $aFileList = _FileListToArray(@WindowsDir, '*.*')
Timer($StopWatch, 'Using original _FileListToArray', True) ; stop timer

;Test _FileListToArrayPlus
Timer($StopWatch) ; start timer
Local $bFileList = _FileListToArrayPlus(@WindowsDir, '*.*')
Timer($StopWatch, 'Using _FileListToArrayPlus', True) ; stop timer

_ArrayDisplay($aFileList, 'From original _FileListToArray')
_ArrayDisplay($bFileList, 'From _FileListToArrayPlus')
Exit

Func _FileListToArrayPlus($sPath, $sFilter = '', $iFlag = 0, $sOrder = '')
   ;   ================================================================
   ;   (Conceptual replacement for _FileListToArray)
   ;
   ;   Description .....
   ;      Lists files and\or folders in a specified path (actually uses Dir /B)
   ;
   ;   Syntax ..........
   ;      _FileListToArrayPlus($sPath[, $sFilter = ''[, $iFlag = 0[, $sOrder = 0]]])
   ;
   ;   Parameters ......
   ;      $sPath   - Path to generate filelist for.
   ;      $sFilter - Optional the filter to use (same syntax as DIR).
   ;      $iFlag   - Optional: specifies results to return (add values to combine).
   ;         |0 Return both files and folders (Default)
   ;         |1 Return files only
   ;         |2 Return Folders only
   ;         |4 Return full-path
   ;         |8 Recurse sub-folders
   ;      $sOrder  - Optional: specifies how to sort returned files/folders.
   ;         N  By name (alphabetic)       S  By size (smallest first)
   ;         E  By extension (alphabetic)  D  By date/time (oldest first)
   ;         G  Group directories first    -  Prefix to reverse order
   ;            eg. "GN" for folders first then alphabetically by name
   ;
   ;   Return values ...
   ;      One-dimensional array made up as follows:
   ;         $array[0] = Number of Files\Folders returned
   ;         $array[1] = 1st File\Folder
   ;         $array[2] = 2nd File\Folder
   ;         $array[3] = 3rd File\Folder
   ;         $array[n] = nth File\Folder
   ;
   ;      @Error -
   ;            |1 = Path not found or invalid
   ;            |2 = Invalid $sFilter
   ;   ================================================================
   Local $i, $foo, $Line, $aFileList, $sRecurse, $sAttrib = '/a'
   $sPath = NoBackslash(StringRegExpReplace($sPath, '[\\/]+\z', ''))
   If Not FileExists($sPath) Then Return SetError(1, 1, '')
   If StringRegExp($sFilter, '[\\/:><\|]|(?s)\A\s*\z') Then Return SetError(2, 2, '')
   If $sFilter = '*' Or $sFilter = '*.*' Then $sFilter = ''
   If $sFilter <> '' Then $sFilter = '\' & $sFilter
   If Not (BitAND($iFlag, 1) = 1 And BitAND($iFlag, 2) = 2) Then ; in case $iFlag is 3, 7, 11, or 15 (assumes both files and folders)
      If BitAND($iFlag, 1) = 1 Then $sAttrib &= '-D' ;files only
      If BitAND($iFlag, 2) = 2 Then $sAttrib &= 'D' ;folders only
   EndIf
   If BitAND($iFlag, 8) = 8 Then $sRecurse &= '/s' ;recurse
   $sOrder = StringReplace(StringReplace($sOrder, '/o:', ''), '/o', '')
   If $sOrder <> '' Then $sOrder = '/o' & $sOrder
   $foo = Run(@ComSpec & ' /c DIR "' & $sPath & $sFilter & '" /b' & $sAttrib & $sRecurse & $sOrder, '', @SW_HIDE, $STDOUT_CHILD)
   Do
      $Line &= StdoutRead($foo)
   Until @error
   $aFileList = StringSplit(StringLeft($Line, StringLen($Line) - 2), @CRLF, 1)
   If @error = 1 Then
      ReDim $aFileList[1]
      $aFileList[0] = 0
   ElseIf BitAND($iFlag, 8) <> 8 And BitAND($iFlag, 4) = 4 Then ; add full-path
      For $i = 1 To $aFileList[0]
         $aFileList[$i] = $sPath & '\' & $aFileList[$i]
      Next
   EndIf
   Return $aFileList
EndFunc

Func Timer(ByRef $Timer, $Message = '', $ShowTicks = False)
   ;   ===================================================
   ;   Parameters ....
   ;      $Timer      - The timer object to use.
   ;      $Message   - Optional: leave empty to only (re)start $Timer
   ;      $ShowTicks - Optional: True to not convert to HH:MM:SS format
   ;
   ;   Example of use:
   ;      Timer($StopWatch) ; start timer
   ;      Timer($StopWatch, "Post-processing done!", True) ; stop timer
   ;   ===================================================
   Local $Secs, $Mins, $Hour, $Time = Int(TimerDiff($Timer))
   If Not $ShowTicks Then
      _TicksToTime($Time, $Hour, $Mins, $Secs)
      $Time = StringFormat('%02i:%02i:%02i', $Hour, $Mins, $Secs)
   EndIf
   If $Message <> '' Then MsgBox(262144, $Message, $Time)
   $Timer = TimerInit()
EndFunc   ;==>Timer


[The Freezer]

So far my tests are showing an increase in speed of between 30-40%.

EDIT: Appears to be for recursive listings only though. It's actually a little slower than the original otherwise (likely due to taking an overhead cost with the external call).

[bphlpt]

Even better than advertised! Wonderful! A success then I assume?

Just curious - I can't see how $aFileList[0] is assigned the count of how many elements are added to the array, in fact I never see the count get created at all. Where does that happen? Also, are there any differences in result between your revised code and the original much, much longer AutoIt script? Maybe exit error codes? I guess you might want to share this back with Onepiece, and even the AutoIt community?

Cheers and Regards

EDIT:

If BitAND($iFlag, 1) = 1 Then $parm &= ' /a:-D' ;files only
If BitAND($iFlag, 2) = 2 Then $parm &= ' /a:D' ;folders only

This looks like you could set both of these flags, and that doesn't make sense, does it? Shouldn't you only be able to set one or the other, or do you set both if you want both files and folders?

[The Freezer]

Even better than advertised! Wonderful! A success then I assume?

Partially. It seems the results greatly improve with increased number of files/folders to collect -- for example, from the "C:\Windows" folder. But the results actually get worse with much smaller numbers such as from the "D:\ppApps" folder (up to 10 times slower but entirely negligible when dealing with such few numbers). I think this might be caused by a couple things. The overhead involved in calling out with the RUN function and the fact that DIR will blindly catalog everything to the fullest depths whereas I can code in my recursion to skip going any deeper based on certain assumptions about SetupS apps/games.

• @WindowsDir
  _FileListToArray = 3000 ms
  _FileListToArrayPlus = 1800 ms

• x:\ppApps
  _FileListToArray = 25 ms (with "recursion shortcuts")
  _FileListToArray = 200 ms (without "recursion shortcuts")
  _FileListToArrayPlus = 250 ms

As you can see, with the first example (even with "recursion shortcuts"), that we have a savings of almost 1.5 seconds but in the second example, at worse we're looking at a loss of about 0.2 seconds.
__________________________________________

Just curious - I can't see how $aFileList[0] is assigned the count of how many elements are added to the array, in fact I never see the count get created at all. Where does that happen?

Checkout the "StringSplit" function.

Code: Select all

Returns an array, by default the first element ($array[0]) contains the number of strings returned, the remaining elements ($array[1], $array[2], etc.) contain the delimited strings.

__________________________________________

Also, are there any differences in result between your revised code and the original much, much longer AutoIt script? Maybe exit error codes? I guess you might want to share this back with Onepiece, and even the AutoIt community?

I added a couple parameters to allow for a more direct substitution of the _FileListToArray function with little or no modification of the surrounding code. It's far from complete and perhaps later I'll make it a more complete implementation (blackbox) of it but for now it suits my needs
__________________________________________

If BitAND($iFlag, 1) = 1 Then $parm &= ' /a:-D' ;files only
If BitAND($iFlag, 2) = 2 Then $parm &= ' /a:D' ;folders only

This looks like you could set both of these flags, and that doesn't make sense, does it? Shouldn't you only be able to set one or the other, or do you set both if you want both files and folders?

True if you passed "3" for $iFlag; but you're supposed to set it to "0" to get both files and folders. Besides, that's how the original _FileListToArray does it and I would therefore need to "emulate" that behavior as well. From the Help-file for _FileListToArray:

Code: Select all

$iFlag: [optional] specifies whether to return files folders or both
$iFlag=0(Default) Return both files and folders
$iFlag=1 Return files only
$iFlag=2 Return Folders only

Note, in my implementation you would set it to "4" to get both files and folders recursively (rather than trying to use "7").

Later I'll add another $iFlag value "8" for full path to make it more "compatible" with the original _FileListToArray. It didn't recurse thru subfolders and therefore never returned the full path (strictly just the file/folder names). I usually had to follow behind it with a for-next loop to add in the full path.

[bphlpt]

Yeah, your method above to add full path only works if you are not being recursive. You would think that Dir would have a flag option to output the fully qualified path, similar to the parameter extension %~f1, I mean it knows it as it's printing out it's list. It sure would make things easier. Otherwise, I guess the quickest way might be to write a batch recursive routine, remember I've proved it can be done easily enough, but then you would get into the overhead of multiple Dir calls, so it might end up that the existing _FileListToArray might be just as fast. hmmmm

Cheers and Regards

[The Freezer]

To be fair, _FileListToArray mentions that it is basically the DIR /b and when you think about it that if you're not doing recursive listings then you probably already would (or should) know the path you're getting the listing for anyway.

Since it would make no sense or I don't think anything useful can be done to recursive listings without full-paths and also the original _FileListToArray doesn't have a recursive setting for $iFlag, I think it would simply just need to be changed like so:

Code: Select all

ElseIf BitAND($iFlag, 4) <> 4 And BitAND($iFlag, 8) = 8 Then ; add full-path

So except for returning the same error codes, this would make it nearly identical to the original _FileListToArray except this new one is more capable now.

EDIT: Revised first post to reflect my most recent changes to the function

[bphlpt]

It ends up that Dir is smarter than I gave it credit for. It agrees with you that nothing useful can be done with recursive listings without full-paths. If you specify the /s option it automatically outputs in full paths, in fact I don't think there's anyway to NOT get full paths with recursive listings. So you only have to add paths, if you want them, if you are NOT doing a recursive list, which is what I think you ended up with in your script. Also you might want to check your sort order option. At least in Win7 x64, /ong gives different results than /ogn. I think you want /ogn. And lastly, you might want to include /a, "blank" with no modifiers, when you want a listing with both directories and files. I read something about that being a better option and I THINK some of my tests confirmed that, but then I couldn't seem to repeat those results. Anyway, if it doesn't do any harm, and doesn't take any extra time, it might be a good idea.

Cheers and Regards

[The Freezer]

Also you might want to check your sort order option. At least in Win7 x64, /ong gives different results than /ogn. I think you want /ogn.

Yeah, it looks like /ong gives the exact same results as /on regardless (basically the 'n' ignores the 'g'). Whereas /og just separates folders from files without sorting alphabetically; and /ogn separates then sorts alphabetically within each group.

On a related note, something I had not noticed before was that with _FileListToArray the resulting array is unsorted (just as if the /o is never used). This would impact the order that SetupS processes FAT drives -- as it does with ssWPI (or at least it used to). For example, one needs DotNET or Java installed before an app that requires it. To fix this with SetupS I would need to add the _ArraySort function after using _FileListToArray -- or use the new FileLIstToArray

EDIT: Turns out I had already been using _ArraySort to sort results from _FileListToArray -- even those from NTFS disks that didn't need it. So now it only does an _ArraySort only if the drive is a FAT one.

Since sorting is not an option with _FileListToArray I'll be adding an extra parameter called $sOrder that uses the same sort options as DIR.

And lastly, you might want to include /a, "blank" with no modifiers, when you want a listing with both directories and files. I read something about that being a better option and I THINK some of my tests confirmed that, but then I couldn't seem to repeat those results. Anyway, if it doesn't do any harm, and doesn't take any extra time, it might be a good idea.

Yep, /a captures all files with any attribute (including Hidden/System). Without it those don't get listed. So definitely better to have it than not.

The following is a listing from a FAT drive with some test files on it. "H.Test.1.avi" has the hidden attribute and "S.Test.2.avi" has the System one. Also, the "Recycled" folder has a System attribute.

Using "Dir /b/a" (Also, notice the seemingly "randomness" of FAT-drive listings):

Code: Select all

Z.Test.4.avi
Folder.Test
A.Folder.Test
Recycled
S.Test.2.avi
A.Test.3.avi
H.Test.1.avi


"Dir /b"

Code: Select all

Z.Test.4.avi
Folder.Test
A.Folder.Test
A.Test.3.avi


"Dir /b/a/on"

Code: Select all

A.Folder.Test
A.Test.3.avi
Folder.Test
H.Test.1.avi
Recycled
S.Test.2.avi
Z.Test.4.avi


"Dir /b/a/ong" (Notice it ignored 'g')

Code: Select all

A.Folder.Test
A.Test.3.avi
Folder.Test
H.Test.1.avi
Recycled
S.Test.2.avi
Z.Test.4.avi


"Dir /b/a/og"

Code: Select all

Folder.Test
A.Folder.Test
Recycled
Z.Test.4.avi
S.Test.2.avi
A.Test.3.avi
H.Test.1.avi


"Dir /b/a/ogn"

Code: Select all

A.Folder.Test
Folder.Test
Recycled
A.Test.3.avi
H.Test.1.avi
S.Test.2.avi
Z.Test.4.avi


_________________________________

Here's another thought... could it be that the DIR sorting is causing the slight overhead versus the _FileListToArray since it doesn't sort its listings?

[bphlpt]

With all of these improvements, if you add the necessary code to produce the same return error codes as _FileListToArray does, if you feel that's at all important, then it sure seems to me that your improved version is much more capable and only sightly slower under only a very few circumstances. Enough so that I would think your new version can completely replace the use of _FileListToArray. Can you think of any circumstances under which it would be better to use the original _FileListToArray, either because of noticeable speed differences or because of a difference of actions? If so, then I guess you might could test for those circumstances and call _FileListToArray from within your new routine, if those tests were able to be done without actually doing the function anyway AND if the tests didn't take longer than the time savings you were wanting to get.

As to the sorting difference causing the extra overhead - perhaps. And since I would think that sorting would be desirable more often than not, then timing tests should probably include a sorting loop when timing _FileListToArray.

Great work on fine tuning this routine!

Cheers and Regards

[The Freezer]

Did some more fine-tuning on this function. And revised the first post to reflect my most recent changes.

Some notes:

• Swapped $iFlag values for return Full-path (now "4" to set) and Recursion ("8" to set).

• Keeping the same syntax for sorting as the DIR command. Much more concise, I believe.

• Added some of the original's error codes.
  
  • Dropped @error = 3 (Invalid $iFlag) because the new method handles those flags better than the original.
  • Dropped @error = 4 (No File(s) Found) in favor of the following change from the original...

• If no files or folders found then the number of files\folders returned ($array[0]) will be zero.

[The Freezer]

Can you think of any circumstances under which it would be better to use the original _FileListToArray, either because of noticeable speed differences or because of a difference of actions? If so, then I guess you might could test for those circumstances and call _FileListToArray from within your new routine, if those tests were able to be done without actually doing the function anyway AND if the tests didn't take longer than the time savings you were wanting to get.

Yes, I think so. It seems the ppApps/ppGames benefit most from the "old" method (with the shortcuts) and %ProgramFiles% and %SystemRoot% benefit from the "new" method. And that is easy to test for.

In fact, I've incorporated this new-new method into RC5 and tests seem to be as expected so far...