crawler: inconsistent data from different servers

[wander]

Despite identical SOA serial the servers for com.br seem to deliver inconsistent NSEC records:

#?>!..#?=!..#?=!..#?.<!.#?.<!.#?.>!.#?.<!.#?>!..#?=!..?=!..#?
Inconsistent cut with serial 2013052006 FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI
Removing old NSEC FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U oldserial 2013052006 newserial 2013052006
Replacing 1 ranges with gap
Cutting 2 gaps with FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI
Removing (NSECName(FJ320K92T96U622KMLPLI8KRC2OQH838), NSECName(FJ344Q41S6Q2UR5030B45N82SL182QSM))
Right cut (NSECName(FJ344Q41S6Q2UR5030B45N82SL182QSM), NSECName(FJ3I66HU4M31SQ64J19F1UHDLRH5682U))
!..#?>!..#?.<!.#?.<!.#?.|!.#?.<!.#?|!..#?|!..?|!..#?|!..#?
Inconsistent cut with serial 2013052006 FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U
Removing old NSEC FJ320K92T96U622KMLPLI8KRC2OQH838 FJ3524MSF0JTG389DUBHF30T6AHU0POI oldserial 2013052006 newserial 2013052006
Replacing 1 ranges with gap
Cutting 2 gaps with FJ344Q41S6Q2UR5030B45N82SL182QSM FJ3I66HU4M31SQ64J19F1UHDLRH5682U
Left cut (NSECName(FJ320K92T96U622KMLPLI8KRC2OQH838), NSECName(FJ3524MSF0JTG389DUBHF30T6AHU0POI))
Removing (NSECName(FJ3524MSF0JTG389DUBHF30T6AHU0POI), NSECName(FJ3I66HU4M31SQ64J19F1UHDLRH5682U))

• Notify operator of possible fault?
• Add feature to prefer smaller (more specific) NSEC3 range in case of identical SOA serial?

[wander]

As noted in #23, servers may serve different SOA serials with inconsistent data. Example:

opening file ../../domaindata/domain_xn--mgbx4cd0ab.db
?Lucky hit: xn--mgbx4cd0ab.
?Lucky hit: xn--mgbx4cd0ab.
Assuming closest encloser xn--mgbx4cd0ab. -> 5RI9C0D5IMPC1OGKGR88LQNNNRK59LDS
Assuming wildcard *.xn--mgbx4cd0ab. -> 3COM8GJ78BEGD0HM8I27AMEF15390JS5
paramset: ParamSet[NSEC3, Algorithm=1, Flags=0, Iterations=10, Salt=eb9717
paramid: 2
Initializing Gaps for paramid 2
No ranges in database yet
Fresh run
?Old: ParamSet[NSEC3, Algorithm=1, Flags=0, Iterations=10, Salt=eb9717
New: ParamSet[NSEC3, Algorithm=1, Flags=1, Iterations=10, Salt=a1699e