Opened 10 years ago

Last modified 10 years ago

#39 new defect

Add support for wildcarded domains

Reported by: wander Owned by:
Priority: important Component: crawler
Keywords: Cc:

Description

Example:

plindex.com. 1401573062.87
Got IP for ns1.transip.nl.: 80.69.69.69
Got IP for ns1.transip.nl.: 2a01:7c8:b::53
Got IP for ns0.transip.net.: 80.69.67.67
Got IP for ns0.transip.net.: 2a01:7c8:a::53
Got IP for ns2.transip.eu.: 217.115.203.194
Got IP for ns2.transip.eu.: 2001:14a0:100:6::53
Connecting to database...
?Lucky hit: plindex.com.
?Lucky hit: plindex.com.
Assuming closest encloser plindex.com. -> EH7DC386J9T5AKS4VA1EEII8CCNBN67T
Assuming wildcard *.plindex.com. -> K06490OBB70PKKMD3N38JRJB1PISSH8R
paramset: ParamSet[NSEC3, Algorithm=1, Flags=None, Iterations=100, Salt=77a6ea5cde8c8255
paramid: 5602
Loading missing Gaps for paramid=5602 from database...
No ranges in database yet
Fresh run
?Lucky hit: vh6e2g04fh.plindex.com.
Traceback (most recent call last):
  File "crawler.py", line 1040, in <module>
    freshrun()
  File "crawler.py", line 614, in freshrun
    (serial, nseclist) = extract_nsec(response)
  File "crawler.py", line 742, in extract_nsec
    assert serial is not None, "SOA not found"
AssertionError: SOA not found

Change History (3)

comment:1 Changed 10 years ago by wander

Component: nsec3breakercrawler

comment:2 Changed 10 years ago by wander

Might be related or a broken zone:

ander@crawler:~/nsec3breaker$ python crawler.py lanserius.com
Got IP for ns1.loopia.se.: 93.188.0.20
Got IP for ns2.loopia.se.: 93.188.0.21
Connecting to database...
?Lucky hit: lanserius.com.
?Lucky hit: lanserius.com.
Assuming closest encloser lanserius.com. -> QP3DE8F8AAOCSQ61U69IN0HAS9J0NHB2
Assuming wildcard *.lanserius.com. -> HC499OJ6F80ETL1VCL0NB1VN8CDLTC7J
paramset: ParamSet[NSEC3, Algorithm=1, Flags=None, Iterations=1, Salt=ab
paramid: 11699
Loading missing Gaps for paramid=11699 from database...
Testing whether old guesses cut current Gaps...

Hashed 0 old guesses, switching to regular mainloop
#BUILD STATUS

?NODATA for: 39035rzj4z7.lanserius.com.
.|!.?NODATA for: 39035r3wklm.lanserius.com.
.|!.2229 gaps in memory for lanserius.com.
SQL COMMIT
#?NODATA for: oyxz47qec9l.lanserius.com.
.|!.#?NODATA for: ebr2ozelply.lanserius.com.
.|!.?NODATA for: ebr2ozjx361.lanserius.com.
.|!.#?NODATA for: ouaewh52t8l.lanserius.com.
.|!.?NODATA for: ouaeuui2ptv.lanserius.com.
.|!.#?NODATA for: xg75u2x2mhr.lanserius.com.
.|!.?NODATA for: xg75u22e021.lanserius.com.
.|!.#?NODATA for: vrtlsvc3d5i.lanserius.com.
.|!.#^Cgot ctrl+c
?NODATA for: dd7k33zmk46.lanserius.com.
.|!.?NODATA for: dd7k2gbmgql.lanserius.com.
[pressed ctrl+c]
.|!.Progress: 0.0 %
Crawled 2238 out of estimated 0 hashes
2239 gaps in memory for lanserius.com.
SQL COMMIT

comment:3 in reply to:  2 Changed 10 years ago by wander

Replying to wander:

Might be related or a broken zone:
[...]

Nope, minimally covering NSEC3 records (white lies). Follow-up as #40.

Note: See TracTickets for help on using tickets.