HEX Pattern searching using TC

English support forum

Moderators: white, Hacker, petermad, Stefan2

devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

milo1012 wrote:So is your binary file in fact a doc, odt, pdf, rtf, sxw or xml file? In that case you'd need to disable the filter to make binary search possible.
none of the mentioned above
but if I want it disabled anyway, how do I do it ?
milo1012 wrote:If this is still not helping, I can only guess that your PCREsearch settings don't fit. You may post the content of your PCREsearch.Sample.ini/PCREsearch.ini somewhere (either directly here as a code block if it's not too big, or on some extern site and link to it).
can't post A URL in any way, but I think its only for the first day, will try tomorrow.

An interesting thing I've just noticed.

the file(s) in question (I'm trying to search for(/in)) are very large files,
I've just created a few small file with that pattern and searched with same exact settings made, and it worked just fine (for the small example files)
but when trying to make it real, and search the actual (large) files
it doesn't work.
what can it be ?
does it has something to do because of being large file(s) ?
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

devd wrote:the file(s) in question (I'm trying to search for(/in)) are very large files,
Ah, that might be the culprit. The default limits are 50 MiB for custom columns, which you can only bypass with <spacebar> or the TC commands cm_LoadAllOnDemandFields and cm_LoadSelOnDemandFields.
The 2nd limit is the global file size limit which is 2GiB (2048 MiB) by default.
Just use PCREsearchConfig.exe and set one or both entries to a higher value.
(note that I added these limits as a result of an unresponsive TC when searching large files for content, I also covered this in the readme file)
devd wrote:but if I want it disabled anyway, how do I do it ?
Fastest way: use PCREsearchConfig.exe - in the lower right corner you will have the filter extension setting. Just remove the fitting file extension entries in the dropdown list where you don't want a text filter (button "-")
TC plugins: PCREsearch and RegXtract
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

milo1012 wrote:The 2nd limit is the global file size limit which is 2GiB (2048 MiB) by default.
the files searched in are more than double in size
does it mean nothing can be done ?
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

devd wrote:
milo1012 wrote:The 2nd limit is the global file size limit which is 2GiB (2048 MiB) by default.
the files searched in are more than double in size
does it mean nothing can be done ?
Like I said: just set it to a higher value.
Use PCREsearchConfig.exe and set the "Global" dropdown box (in the "File size limits" section) to sth. like "999999" (you can edit the field, not just select from the dropdown list) and it should work.
TC plugins: PCREsearch and RegXtract
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

Yes, just edited the INI file manually, and it seems to work now
thanks A lot

another question
can I use wildcards in it, like:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{[10-100]}\x1f\x88
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

devd wrote:can I use wildcards in it, like:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{[10-100]}\x1f\x88
No, this won't work, but this would:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}\x1f\x88
Explanation:
PCRE syntax wrote:QUANTIFIERS
? 0 or 1, greedy
?+ 0 or 1, possessive
?? 0 or 1, lazy
* 0 or more, greedy
*+ 0 or more, possessive
*? 0 or more, lazy
+ 1 or more, greedy
++ 1 or more, possessive
+? 1 or more, lazy
{n} exactly n
{n,m} at least n, no more than m, greedy
{n,m}+ at least n, no more than m, possessive
{n,m}? at least n, no more than m, lazy
{n,} n or more, greedy
{n,}+ n or more, possessive
{n,}? n or more, lazy
TC plugins: PCREsearch and RegXtract
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

thank you very much you have been very helpful
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

Oh, one more thing please

is there a way to use "or"
Example, search in A file this:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}\x1f\x88
"or" this:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}\x2b\xed
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

devd wrote:is there a way to use "or"
Example, search in A file this:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}\x1f\x88
"or" this:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}\x2b\xed
Yes, there are many possibilities for this in RegEx. Sth like this should work:

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}(?:\x1f\x88|\x2b\xed)
And BTW, no need to set the full range of hex bytes each time, you can use the dot/period placeholder instead:

Code: Select all

\x00\x00\x46\x49\x4c\x45.{10,100}(?:\x1f\x88|\x2b\xed)
TC plugins: PCREsearch and RegXtract
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

great, thank you
User avatar
Stefan2
Power Member
Power Member
Posts: 4126
Joined: 2007-09-13, 22:20 UTC
Location: Europa

New user are not allowed to post valid URLs Links

Post by *Stefan2 »

devd wrote: can't post A URL in any way,
New user are not allowed to post valid URLs
viewtopic.php?p=339358#339358
to prevent spam.


But of course you can post plain text
https: ghisler.ch/board/viewtopic.php?p=339358#339358








 
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

I'm getting to many results, and need to narrow it down a bit further

I need to add to end

*(\x01\x02 or \x31\x38)\x31\x36

is the code below correct ?

Code: Select all

\x00\x00\x46\x49\x4c\x45[\x00-\xff]{10,100}\x2b\xed[\x00-\xff]{10,20}(\x01\x02|\x31\x38)\x31\x36
really appreciate it
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

devd wrote:is the code below correct ?
Yes, seems correct.
BTW, there is a built-in RegEx syntax/error checker in the config tool.
You can also use online tools like https://regex101.com.
TC plugins: PCREsearch and RegXtract
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

milo1012 wrote:
devd wrote:is the code below correct ?
Yes, seems correct.
Thanks again
milo1012 wrote:BTW, there is a built-in RegEx syntax/error checker in the config tool.
You can also use online tools like https://regex101.com.
Great, that's good, very good info
devd
Junior Member
Junior Member
Posts: 21
Joined: 2018-03-28, 12:23 UTC

Post by *devd »

finally got to that part
milo1012 wrote:You can even get the file offset(s):
Change field type to "String output".
Replace String:

Code: Select all

\xO0, 
very interesting, the code I use is working just fine
but when added/changed to "String output", I'm getting no results

EDIT: do I need to change anything in "OP" or "value"
I've tried every possibility in "OP" but maybe need to add something in "VALUE"

Image: http://i64.tinypic.com/sc6gwn.jpg

milo1012 wrote:You can also use online tools like https://regex101.com.
is there a way I can use Hex in that online tool ?
I tried, but it interpret it as text
Post Reply