wiki:CrawlerExample

Crawling mil

$ python crawler.py mil
Got IP for EUR2.NIPR.mil.: 199.252.143.234
Got IP for PAC1.NIPR.mil.: 199.252.180.234
Got IP for PAC2.NIPR.mil.: 199.252.155.234
Got IP for EUR1.NIPR.mil.: 199.252.154.234
Got IP for CON2.NIPR.mil.: 199.252.162.234
Got IP for CON1.NIPR.mil.: 199.252.157.234
Connecting to database...
?Lucky hit: mil.
?Lucky hit: mil.
Assuming closest encloser mil. -> 7GBLJ4F0H31BHS1JSPQ4T588RJ3OOHTS
Assuming wildcard *.mil. -> PS4N9NTRKV7GKNODADGLIN99UVU539T5
paramset: ParamSet[NSEC3, Algorithm=1, Flags=None, Iterations=10, Salt=ebfc
paramid: 1794
Loading missing Gaps for paramid=1794 from database...
No ranges in database yet
Fresh run
?!!!Progress: 2.16 %
Crawled 3 out of estimated 138 hashes
0 gaps in memory for mil.
SQL COMMIT
Testing whether old guesses cut current Gaps...

Hashed 0 old guesses, switching to regular mainloop
#BUILD STATUS

?.|!.?.|!.,#?.|!.?.|!.#?.|!.?.|!.#?.|!.?.|!.#?.|!.?.<!.#?.|!.#?.|!.?.|!.?.|!.#?.|!.?.|!.#?.|!.?.<!.?.|!.#?.<!.#?.|!.#?.>!.#?.|!.?.|!.#?.|!.#?.|!.#?.|!.#?.>!.?.>!.#?.<!.?.|!.#?.|!.#?.|!.?.|!.#?.<!.#?.|!.#?ERROR: timeout at attempt=1 for server=199.252.154.234, retrying
?.>!.#?.>!.#?.>!.#?.|!.?.|!.#?.|!.?.|!.#?.>!.#?.>!.?.|!.?.>!.#?.|!.#?.|!.#?.|!.#?.<!.#?.|!.41 gaps in memory for mil.
SQL COMMIT
#?.|!.#?.|!.#?ERROR: timeout at attempt=1 for server=199.252.180.234, retrying
?.=!.#?.>!.?.|!.?.|!.?ERROR: timeout at attempt=1 for server=199.252.155.234, retrying
?.|!.#?.|!.#?.|!.#?.<!.?.>!.#?.|!.?.|!.?.|!.#?.|!.?.>!.?.<!.#?.|!.?.|!.#?.<!.?.|!.#?.>!.?.|!.?.=!.#?.|!.#?.=!.?.<!.#?.<!.#?.|!.?.>!.?.|!.#?.>!.?.|!.#?.|!.?.|!.59 gaps in memory for mil.
SQL COMMIT
#?.>!.?.>!.#?.|!.?.<!.?.>!.#?.|!.#?.<!.?.|!.?.|!.?.=!.#?.|!.#?.>!.?.>!.,#?.|!.?.|!.#?.|!.?.=!.#?.<!.?.>!.?.|!.,#?.>!.?.<!.?.>!.?.<!.#?.>!.?.|!.#?.=!.?.<!.?.|!.?.>!.#?.=!.?.<!.?.<!.#?.=!.?.>!.#?.>!.?.<!.?.=!.#?.=!.?.|!.?.>!.#?.=!.?.>!.#?.<!.?.>!.?.|!.#?ERROR: timeout at attempt=1 for server=199.252.162.234, retrying
?.|!.?.>!.?.=!.#?.<!.?.=!.?.=!.?.<!.#?.<!.?.=!.#?.|!.#?.=!.?.>!.61 gaps in memory for mil.
SQL COMMIT
#?.<!.?.=!.?.=!.#?.>!.,#?.>!.?.=!.?.|!.#?.=!.?.=!.?.=!.?.=!.#?.=!.?.=!.?.<!.#?.=!.?.=!.?ERROR: timeout at attempt=1 for server=199.252.157.234, retrying
?.=!.?.=!.#?.=!.?.=!.?.|!.#?.<!.?.=!.?.=!.#?.<!.?.=!.?.=!.#?.=!.?.=!.#?.>!.?.=!.?.|!.?.>!.#?.=!.?.=!.?.=!.,#?.>!.?.Found wraparound NSEC3 RR, reloading gaps from DB...
!.?.=!.?.<!.#?.=!.?.=!.?.>!.?.=!.#?.<!.?.=!.?.>!.?.=!.#?.=!.?.<!.?.=!.?.=!.#?.<!.?.<!.?.=!.?.<!.#?.=!.?.=!.?.=!.?.=!.#?.=!.?.=!.?.=!.,#?.>!.?.=!.?.>!.?.=!.#?.=!.?.=!.?.|!.?.=!.#?.=!.,?.=!.?.=!.#?.=!.?.=!.,,#?.=!.?ERROR: timeout at attempt=1 for server=199.252.154.234, retrying
?.=!.?.=!.?.=!.7 gaps in memory for mil.
SQL COMMIT
#?.=!.,,#?.=!.?ERROR: timeout at attempt=1 for server=199.252.155.234, retrying
?.=!.,#?.=!.#?.=!.#?.=!.#?.=!.Progress: 100.0 %
Crawled 235 out of estimated 235 hashes
0 gaps in memory for mil.
SQL COMMIT

Handling inconsistencies

Inconsistent cut with serial 1399574412 DITK318N67FA85HE1TKAEVIOC5SST0LC DJ842U70H1OPD3QMBCQQKUN9GN786QTV
Removing old NSEC DITK318N67FA85HE1TKAEVIOC5SST0LC DJ1DA3FUH7VI2SSTKH47EMR69SVI0FDQ oldserial 1399427048 newserial 1399574412
Replacing 1 ranges with gap
Cutting 2 gaps with DITK318N67FA85HE1TKAEVIOC5SST0LC DJ842U70H1OPD3QMBCQQKUN9GN786QTV
Removing (NSECName(DITK318N67FA85HE1TKAEVIOC5SST0LC), NSECName(DJ1DA3FUH7VI2SSTKH47EMR69SVI0FDQ))
Removing (NSECName(DJ1DA3FUH7VI2SSTKH47EMR69SVI0FDQ), NSECName(DJ842U70H1OPD3QMBCQQKUN9GN786QTV))


Inconsistent cut with serial 1399574772 NQ4F5CU1UC1AI9L7A8I3L996N6B8R8V0 NQRUAF1GUBIR2H45192SAS1RKOUBTO8D
Removing old NSEC NPRMCEO5OLQFBH5NU4N6EL3EDMQBL62L NQR7CD5JV2KRNBOJBRF5A3IG4J3T13O5 oldserial 1396270055 newserial 1399574772
Replacing 1 ranges with gap
Cutting 2 gaps with NQ4F5CU1UC1AI9L7A8I3L996N6B8R8V0 NQRUAF1GUBIR2H45192SAS1RKOUBTO8D
Left cut (NSECName(NPRMCEO5OLQFBH5NU4N6EL3EDMQBL62L), NSECName(NQR7CD5JV2KRNBOJBRF5A3IG4J3T13O5))
Right cut (NSECName(NQR7CD5JV2KRNBOJBRF5A3IG4J3T13O5), NSECName(NQSTUJ6DGDHU8815OA478FOTJGQIVAR5))


Inconsistent cut with serial 1399575126 VPI26FMFGRTCCOSUGLVNV43PRND1A9UI VPMS6E3AI6J3MPB87BCEISFNB8F59B5E
Removing old NSEC VPKQU27DOF9K1TC67Q0R84GMJP6H6DEG VPO1BL42H5U5TGSL1J8J80V6MUCA47Q3 oldserial 1396270055 newserial 1399575126
Replacing 1 ranges with gap
Cutting 2 gaps with VPI26FMFGRTCCOSUGLVNV43PRND1A9UI VPMS6E3AI6J3MPB87BCEISFNB8F59B5E
Left cut (NSECName(VP8NQVTNE1KJB5GDGBHL29DTMORDPLCR), NSECName(VPKQU27DOF9K1TC67Q0R84GMJP6H6DEG))
Right cut (NSECName(VPKQU27DOF9K1TC67Q0R84GMJP6H6DEG), NSECName(VPO1BL42H5U5TGSL1J8J80V6MUCA47Q3))


Inconsistent cut with serial 2001525878 SORTK4EU5DHOD04IN7071MTUK9HC3F7N SORV2S57I8I08S9MO94QLARLOSS9PT9G
Removing old NSEC SORTU8G2JO75SI3RDGFV2GVUQ09632V6 SORV2S57I8I08S9MO94QLARLOSS9PT9G oldserial 2001524907 newserial 2001525878
Replacing 1 ranges with gap
Cutting 2 gaps with SORTK4EU5DHOD04IN7071MTUK9HC3F7N SORV2S57I8I08S9MO94QLARLOSS9PT9G
Left cut (NSECName(SORSCK45GH6BMMH890GRS84QLD616TTL), NSECName(SORTU8G2JO75SI3RDGFV2GVUQ09632V6))
Removing (NSECName(SORTU8G2JO75SI3RDGFV2GVUQ09632V6), NSECName(SORV2S57I8I08S9MO94QLARLOSS9PT9G))
Last modified 9 years ago Last modified on 04/05/15 18:53:54