Problems with dots at the end of file names

[MarcinW]

The general rule in Total Commander is: if we try to set the name of a file or a folder with dots at the end (e.g. "test..."), then we get this name with dots truncated (e.g. "test"). It is true when we try to create file a file (Shift+F4), create a folder (F7) or rename a file (Shift+F6). But:

1) renaming a folder by adding only dots (Shift+F6) fails. If we try to rename a folder "test" to "test...", it should be treated as no change and the folder name should be left unchanged as "test". But something strange happens: we get a dialog with the information, that the directory "test.." (one dot less!) already exists. After confirming we get an error dialog.

Very strange things happen when we are inside an archive (tested with a ZIP archive):

2) renaming a FILE (Shift+F6) "test" to "test..." gives "test." (two dots less!)

3) renaming a FOLDER (Shift+F6) "test" to "test..." gives "test.." (one dot less!)

4) but creating a folder (F7) "test..." gives "test..." (all dots as entered)

Moreover (still inside ZIP archive):

5) if we have a FILE "test...", after pressing Shift+F6 the edit field contains "test..." (as it should be)

6) if we have a FOLDER "test...", after pressing Shift+F6 the edit field contains "test.." (one dot less!)

We can't control file/folder names in archives created by tools other than TC, so names with dots at the end should be handled properly. But when we have an archive with the following structure: test.zip\dirname...\file.txt

7) during unpacking archive contents, TC creates (at the local disk) folder "dirname" (without dots), but tries to unpack file.txt to the folder "dirname..." (with dots), so the operation fails.

General observation: there is something, that truncates one dot at the end of folder names - as in 1), 3) and 6) - but not 4).

Regards!

[ghisler(Author)]

This is not caused by TC itself, the Windows functions do this. You can try the same in Explorer, e.g. New -> Folder, name it test... -> the dots are truncated.

Currently I do not have any plans to try to avoid this standard Windows behaviour.

[MarcinW]

Of course you are right. This is the standard Windows behavior and there is no reason to change it. Please note, that my report - besides 1) - applies to archives (ZIP in this example). An archive can be created or designed for non-Windows systems, so dots at the end of file names inside archives should be supported. And TC supports them. My report shows three groups of problems inside archives:

a) with editing names ended with dots (Shift+F6) - edit field gets less dots than name has - see 6),
b) with renaming such names - dots are partially truncated - see 2) and 3),
c) with unpacking from archived folders, that contain dots at the end - see 7).

There is also one problem not connected with archives - see 1).

Regards

[ghisler(Author)]

The problem with one dot less is that the last dot is seen as the separator between name and extension, so it's not usually shown.

[MarcinW]

So checking if the last char is a dot would be an easy (as I think) solution.


BTW: please note, that there are some differences between handling files and folders.

When we have a folder "dir..." inside a ZIP archive, after pressing Shift+F6 we get (in an inplace editor) "dir.." (one dot less), and pressing Enter will give us finally "dir." (two dots less). The editor eats one dot and saving also eats one dot.

For a file "file...", after pressing Shift+F6 we get "file..." (as it should be), but pressing Enter will give us finally "file." (two dots less). The editor doesn't eat any dot, but saving eats two dots simultaneously.

[Horst.Epp]

Stopping to use such strange naming avoids all this problems.
What is your reason for that useless naming ?

[MarcinW]

I don't use such naming, I found this problem by an accident. But some archives may be created by users with OS-es other than Windows, so this naming may be normal for them - they can simply pack existing files or directories with dots at the end (maybe not 10 dots, but just one). So I just reported the problem to the Author, so he can decide what to do with it. However, solution seems to be quite simple.

As Microsoft states, this problem may occur also with spaces - Macintosh users can use (and pack) these files:
http://support.microsoft.com/kb/115827

So - as I just checked - to completely solve this problem, not only dots, but also ending spaces should be handled with care - saved when editing names inside an archive, and removed when unpacking. If we have two files in an archive: "file " (one space) and "file " (two spaces), unpacking both of them currently causes overwriting the first unpacked file with the second, because both of them are being unpacked as "file" (with no dots).

Regards

[Robert040]

Similar issues will occur with mixed case filenames.

Archives that contain files like "Makefile" and "makefile" can't be extracted on Windows. That is not an TC issue.

[MarcinW]

Archives that contain files like "Makefile" and "makefile" can't be extracted on Windows. That is not an TC issue.

That's not true. TC unpacks first of these two files (say "Makefile") and then it tries to unpack second of them (say "makefile"). But it detects that a file with this name already exists (because destination is case-insensitive), so it asks the user what to do: overwrite or not. It's fully proper behavior.

Here there is a test zip file:

Code: Select all

MIME-Version: 1.0
Content-Type: application/octet-stream; name="pack.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="pack.zip"

UEsDBBQAAgAIAEs9OEJ3R9+OBQAAAAoAAAAIAAAAdGVzdC50eHSrqIABAFBLAwQUAAIACACAJThC
jZvVDwUAAAAUAAAACAAAAHRlU1QudHh0Y2DABABQSwECFAAUAAIACABLPThCd0ffjgUAAAAKAAAA
CAAAAAAAAAABACAAAAAAAAAAdGVzdC50eHRQSwECFAAUAAIACACAJThCjZvVDwUAAAAUAAAACAAA
AAAAAAAAACAAAAArAAAAdGVTVC50eHRQSwUGAAAAAAIAAgBsAAAAVgAAAAAA

[MarcinW]

Here is my summary of this thread, after performing some additional investigation.

Some general notes:

Windows doesn't allow spaces or dots at the end of file/folder name (naming conventions tutorial is available here: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx). But: archives, FTPs (handled by Linux servers) and Linux filesystems (available through WFX plugins) can contain such named files/folders. Moreover, TC for Android should properly handle such file/folder names. Moreover, user may enter such names in some dialogs/edit fields. So:
- displaying such names in panels should be properly handled,
- editing such names with Shift+F6 should be properly handled,
- during copying files/folders from sources other than Windows partitions to Windows partitions, name adjustment should be performed - dots and spaces from the end of each part of the path should be removed.

Now some details and some fixes to be done (can be tested inside ZIP archive or with an FTP server). All these problems are connected with others, so I describe all of them together:


Fix 1:
Files (but not folders) ended with dots are being displayed in panels with one dot truncated. Examples: "file..." will be displayed as "file..", "file... " (space at the end) will be displayed as "file.. " or "file.." (I haven't determined it).
Names should be being displayed "as is".


Fix 2:
Shift+F6: Loading file/folder name into the inplace editor should not truncate any dots nor spaces - names should be loaded "as is".
Currently loading a file name truncates one (invisible - see Fix 1) dot, and loading a folder name truncates one dot.
Dots/spaces filtering should be done at the lower level, before executing a name change.
This fix will solve problems described in pt. 5) and 6).


Fix 3:
Shift+F6: Reading file/folder name from the inplace editor (when finishing editing, after pressing Enter) should also not truncate any dots nor spaces - names should be read "as is".
Currently reading a file name truncates one dot (not two - see Fix 1), and reading a folder name truncates one dot.
Dots/spaces filtering should be done at the lower level, before executing a name change.
This fix will solve problems described in pt. 2) and 3).


Fix 4:
Files and folders ended with dots are being shortened by one dot when copying them from inside ZIP archive to an FTP server - but copying them to another ZIP archive will give correct names (without any changes).
For example, copying file "file..." from inside ZIP to an FTP server will give us "file.." (seen as "file." - see Fix 1).
Names should be copied "as is".


Fix 5:
During receiving file/folder names from non-Windows namespace (archive file, FTP server, WFX plugin or user input) to a Windows namespace, file/folder names adjustment should be performed.
This fix will solve problems described in pt. 1) and 7).

Algorithm (let's give it a name: ValidatePath):
Step 1: If path starts with \\.\ (device namespace) - exit with no changes
Step 2: If path starts with \\?\ (extended-length path) - exit with no changes
Step 3: Replace all '/' characters with '\' character (CreateFile also performs this conversion, so we can do it safely - see documentation of CreateFile)
Step 4: Separate (explode) path parts by '\' character
Step 5: For every part: if '.' or '..' then leave this part unchanged, else remove characters at the end - until we reach character other than dot and space. If we get empty string (if there were only spaces and dots) - remove this part of path completely (to avoid converting "c:\dir1\....\dir2\file.txt" into "c:\dir1\\dir2\file.txt" - we should get "c:\dir1\dir2\file.txt")
Step 6: Concat (join, implode) all parts using '\' char as a concating char

Example:
We have: "C:/dir1/dir2 ./file . ."
We replace to: "C:\dir1\dir2 .\file . ."
We separate to: "C:", "dir1", "dir2 .", "file . ."
We remove ending dots and spaces: "C:", "dir1", "dir2", "file"
We concat to: "C:\dir1\dir2\file"

Some valid path examples for tests:

"\\.\COM1"

"\\?\C:\path\file.txt"
"\\?\UNC\server\share\path\file.txt"

"\\server\share\path\file.txt"

"C:\"
"C:\path\file.txt"
"C:file.txt"
"C:.\file.txt"
"C:..\file.txt"
"C:path\file.txt"
"C:.\path\file.txt"
"C:..\path\file.txt"

"\"
"\path\file.txt"
"file.txt"
".\file.txt"
"..\file.txt"
"path\file.txt"
".\path\file.txt"
"..\path\file.txt"

"..\..\file.txt"


Fix 6:
During my tests with a debugger (with TC 8.01 32-bit) i found a function, which is being called before unpacking every file - at address 005A44DC. It checks if the destination file already exists (0041CF0D - call to FindFirstFileW) and then unpacks the file (0041C3BE - call to CreateFileW). But: FindFirstFileW uses "\\?\" prefix, but CreateFileW doesn't use it. Prefix "\\?\" disables any name modifications performed by Windows (see naming conventions tutorial - link above). Let's check:

Code: Select all

CreateFileW('C:\test ',GENERIC_READ or GENERIC_WRITE,0,nil,CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL,0);

will create file "C:\test" (space at the end will be removed),

Code: Select all

CreateFileW('\\?\C:\test ',GENERIC_READ or GENERIC_WRITE,0,nil,CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL,0);

will create file "C:\test " (with space at the end - Windows Explorer can't even open it!).

Now let's say that we have a file "C:\test" and we want to unpack an archive with a file "test " (with space at the end) to "C:\". So, in the first place, FindFirstFileW checks if a file already exists in "C:\". But:

FindFirstFileW("c:\test ") would make an internal conversion to "c:\test" and would find the existing file "c:\test" - result: the overwrite dialog would be displayed.

FindFirstFileW("\\?\c:\test ") will NOT make an internal conversion to "c:\test", it will check for the existence of "c:\test " (space at the end) and will NOT find the existing file "c:\test" - result: no overwrite dialog will be displayed! But CreateFileW doesn't use the "\\?\" prefix, so it will convert "c:\test " to "c:\test" and will overwrite our existing file without displaying the overwrite dialog!

Please note that calling our ValidatePath function in the first place and then adding a "\\?\" prefix and calling FindFirstFileW would be a safe solution. So it's another reason for introducing the ValidatePath function.


Fix 7:
As I described above, firstly FindFirstFileW is being called. If a file hasn't been found, then CreateFileW is being called with CREATE_ALWAYS flag. But this operation is not performed atomically, so it's not safe in the multitasking environment! Between calling FindFirstFileW and CreateFileW, another process may create a file. I haven't disassembled too much code, so I don't know all reasons for using FindFirstFileW. But CreateFileW should be called with flag CREATE_NEW instead of CREATE_ALWAYS, this would prevent from overwriting the file by an unlucky accident - failing of the CreateFileW with the CREATE_NEW flag will indicate that a file appeared in the background.


Regards