HEX Pattern searching using TC
Moderators: white, Hacker, petermad, Stefan2
-
- Junior Member
- Posts: 2
- Joined: 2015-10-01, 07:38 UTC
HEX Pattern searching using TC
Hi,
I need to look for a specific hex pattern in a bunch of files. Pattern looks like
78 56 34 12 ?? ?? ?? 00 04 24 00 21 where ?? can be any hex . How can i do that in TC. Tried replacing ?? by .., ** . Nothing seems to work.
Br;)
I need to look for a specific hex pattern in a bunch of files. Pattern looks like
78 56 34 12 ?? ?? ?? 00 04 24 00 21 where ?? can be any hex . How can i do that in TC. Tried replacing ?? by .., ** . Nothing seems to work.
Br;)
Wildcards won't work in hex search AFAIK, you need to use RegEx.
But:
TC'S RegEx engine is limited, because it doesn't take newline characters into account.
(every line is searched individually, also in binary files)
This means that if your ?? consists of 0x0A, TC won't find it.
Additionally it seems that \x00 doesn't work either.
You may try:
[x] RegEx (2)
But for file listing (F3) you'd have the same problem of course.
To use a precise search, use my PCREsearch (second link above).
Use PCREsearchConfig.exe.
Create a new field, name it to your liking, and insert
RegEx:
Restart TC or use the internal command cm_UnloadPlugins,
and now activate the plug-in field in the search dialog ("Plugins" tab) and start your search.
You can even get the file offset(s):
Change field type to "String output".
Replace String:
You can now create a custom column, with that very field you just created.
In there you can see every file offset (hex) where the string is found, separated by a comma.
But:
TC'S RegEx engine is limited, because it doesn't take newline characters into account.
(every line is searched individually, also in binary files)
This means that if your ?? consists of 0x0A, TC won't find it.
Additionally it seems that \x00 doesn't work either.
You may try:
Code: Select all
\x78\x56\x34\x12[\x00-\xff]{3}.\x04\x24.\x21
But for file listing (F3) you'd have the same problem of course.
To use a precise search, use my PCREsearch (second link above).
Use PCREsearchConfig.exe.
Create a new field, name it to your liking, and insert
RegEx:
Code: Select all
\x78\x56\x34\x12[\x00-\xff]{3}\x00\x04\x24\x00\x21
and now activate the plug-in field in the search dialog ("Plugins" tab) and start your search.
You can even get the file offset(s):
Change field type to "String output".
Replace String:
Code: Select all
\xO0,
In there you can see every file offset (hex) where the string is found, separated by a comma.
TC plugins: PCREsearch and RegXtract
-
- Junior Member
- Posts: 2
- Joined: 2015-10-01, 07:38 UTC
Hi,milo1012 wrote:To use a precise search, use my PCREsearch (second link above).
Use PCREsearchConfig.exe.
Create a new field, name it to your liking, and insert
RegEx:Restart TC or use the internal command cm_UnloadPlugins,Code: Select all
\x78\x56\x34\x12[\x00-\xff]{3}\x00\x04\x24\x00\x21
and now activate the plug-in field in the search dialog ("Plugins" tab) and start your search.
You can even get the file offset(s):
Change field type to "String output".
Replace String:You can now create a custom column, with that very field you just created.Code: Select all
\xO0,
In there you can see every file offset (hex) where the string is found, separated by a comma.
first step says
and I cant find A way to create A new fieldmilo1012 wrote:Create a new field
can someone guide for a noob like me step by step how it's done ?
I tried to follow the above, and got lost in the first step
TIA
Last edited by devd on 2018-03-28, 13:36 UTC, edited 1 time in total.
Fine.devd wrote:Right LOLStefan2 wrote:Actually, 'first step says' : "use my PCREsearch"
yesStefan2 wrote:.... have you installed the mentioned "PCREsearch" first?
( For others > milo1012s' PCREsearch > http://ghisler.ch/board/viewtopic.php?t=38098 )
Launch the "PCREsearchConfig.exe".
I just took a look and it's not very clear what to do there next (even after reading the help).
As for what I see there, the List entry "PCRE-RegEx14" on the left hand seems to be a not already used "field".
See "PCREsearchConfig utility"-section in "TC\plugins\wdx\PCREsearch\Readme\Readme.html"
Use that entry (or the next free one) to create a "new field", an own entry:
- exchange the Regular Expression with the wanted one,
- clear or modify the Replace box,
- as Name enter what you like to indicate this entry.
Select at least "14" at the Fields box in the top left corner.
(or as many you need to get the '-->' mark on your 'devd' entry)
Click at [OK] to close the PCREsearchConfig.exe.
Restart TC or use the internal command cm_UnloadPlugins,
Now activate the plug-in field in the search dialog ("Plugins" tab)
- choose PCREsearch and
- select the Name (Property) you have chosen to indicate the new entry.
HTH?
If you need more info, wait till milo1012 visits the forum, another can help, or search the forum for other examples for this PCREsearch tool.
Thanks
field type needs to be changed ?
(the default is A Boolean)
so far when doesn't chosen a Boolean or Encoding check it doesn't seem to work
what should I choose?
when next opened it appears again ($0\x20)
field type needs to be changed ?
(the default is A Boolean)
so far when doesn't chosen a Boolean or Encoding check it doesn't seem to work
what should I choose?
I have cleared that, and even after savedStefan2 wrote: - clear or modify the Replace box,
when next opened it appears again ($0\x20)
2devd
So you managed to create a new field and a boolean search worked for you, but you want what I described in the 2nd part of my post? In that case you need to do as I said:
Set the field type to "String output" and - most important - set the replace string to
(note that the character trailing the x is not a zero, but the capital letter "O"!)
This would output the file offset(s) at which your search strings are found.
Since it makes no sense to have no replace string at all (for string field types), this will reappear when you cleared it.
So you managed to create a new field and a boolean search worked for you, but you want what I described in the 2nd part of my post? In that case you need to do as I said:
Set the field type to "String output" and - most important - set the replace string to
Code: Select all
\xO0,
This would output the file offset(s) at which your search strings are found.
Yes, this is the default replace string, i.e. all output results are separated by a space character.devd wrote:I have cleared that, and even after savedStefan2 wrote: - clear or modify the Replace box,
when next opened it appears again ($0\x20)
Since it makes no sense to have no replace string at all (for string field types), this will reappear when you cleared it.
TC plugins: PCREsearch and RegXtract
@milo
thanks for helping me
have try to play with it
but either I'm getting all the files in the directory chosen the search (which can't be)
Or I'm getting no founds at all (which is also not true)
need to it work first
thanks for helping me
yes, I have used one the field mentioned by @stefan2milo1012 wrote:So you managed to create a new field
no, its not workingmilo1012 wrote:and a boolean search worked for you,
have try to play with it
but either I'm getting all the files in the directory chosen the search (which can't be)
Or I'm getting no founds at all (which is also not true)
still not there yetmilo1012 wrote:but you want what I described in the 2nd part of my post?
need to it work first
I assume you restarted TC or used cm_UnloadPlugins after changing the regex for the field?devd wrote:...
either I'm getting all the files in the directory chosen the search (which can't be)
Or I'm getting no founds at all (which is also not true)
If this is not the problem, make sure the expression really *should* work, i.e. maybe post the RegEx you are using and what you expect it to match.
TC plugins: PCREsearch and RegXtract
of course, after every change I tried/mademilo1012 wrote:I assume you restarted TC or used cm_UnloadPlugins after changing the regex for the field?
Why notmilo1012 wrote:If this is not the problem, make sure the expression really *should* work, i.e. maybe post the RegEx you are using and what you expect it to match.
but the system does not let me attach anything because of being a new member
Edit: it may let me attach code
will try
Last edited by devd on 2018-03-28, 20:58 UTC, edited 1 time in total.
Oh, it does let me attach code
example of the (part) file tried to search in
RegEx using:
example of the (part) file tried to search in
Code: Select all
000014A0: 00 00 00 00 00 00 00 00|00 00 00 00 00 00 00 00
000014B0: 46 49 4C 45 00 00 00 00|80 00 0E 02 01 00 00 00
000014C0: 00 00 00 00 03 00 00 00|00 00 00 00 00 00 00 00
000014D0: 00 00 00 00 03 00 00 00|00 00 00 00 18 00 68 00
000014E0: 02 00 F7 0E 00 00 00 00|1F 88 F8 29 ED 1F 88 F8
000014F0: 29 EA 00 00 00 00 00 1F|88 F8 29 EA 01 00 00 00
00001500: 01 00 00 00 10 00 58 00|31 00 32 00 33 00 34 00
00001410: 2E 00 74 00 78 00 74 00|80 00 00 00 00 00 00 00
00001420: 00 00 00 00 00 00 00 00|00 00 00 00 00 00 00 00
00001430: 4E 41 43 4C 02 00 20 00|5C 00 00 00 00 00 00 00
00001440: 00 00 00 00 73 0D 01 00|04 80 14 00 00 00 1B EE
Code: Select all
\x00\x00\x46\x49\x4c\x45[\x00-\xff]{52}\x1f\x88
2devd
Strange, it works for me perfectly. After I created a file with your hex sequence and with your RegEx, it will find results in/for that very file only, nothing else.
At first I thought that the hex snippet represents a file in UTF-16, but this is in fact not the case. But maybe your file is somehow caught by the xdoc2txt filter that PCREsearch is using by default. So is your binary file in fact a doc, odt, pdf, rtf, sxw or xml file? In that case you'd need to disable the filter to make binary search possible.
If this is still not helping, I can only guess that your PCREsearch settings don't fit. You may post the content of your PCREsearch.Sample.ini/PCREsearch.ini somewhere (either directly here as a code block if it's not too big, or on some extern site and link to it).
Strange, it works for me perfectly. After I created a file with your hex sequence and with your RegEx, it will find results in/for that very file only, nothing else.
At first I thought that the hex snippet represents a file in UTF-16, but this is in fact not the case. But maybe your file is somehow caught by the xdoc2txt filter that PCREsearch is using by default. So is your binary file in fact a doc, odt, pdf, rtf, sxw or xml file? In that case you'd need to disable the filter to make binary search possible.
If this is still not helping, I can only guess that your PCREsearch settings don't fit. You may post the content of your PCREsearch.Sample.ini/PCREsearch.ini somewhere (either directly here as a code block if it's not too big, or on some extern site and link to it).
TC plugins: PCREsearch and RegXtract