Norconex commons lang

GivenA page linking to a tel: URI: Norconex test Phone Number ">>html lang="en"> head> title>Norconex testtitle> head> body> a href="tel:123">Phone Numbera> body>html>And the following config: ">xml version="1.0" encoding="UTF-8"?>httpcollector id="test-collector"> crawlers> crawler id="test-crawler"> startURLs> url> startURLs> crawler> crawlers>httpcollector>ExpectedThe collector should not follow this link – or that of any other schema it can't actually process.ActualThe collectors tries to follow the tel: link.INFO [SitemapStore] test-crawler: Initializing sitemap store...INFO [SitemapStore] test-crawler: Done initializing sitemap store.INFO [HttpCrawler] 1 start URLs identified.INFO [CrawlerEventManager] CRAWLER_STARTEDINFO [AbstractCrawler] test-crawler: Crawling references...INFO [CrawlerEventManager] DOCUMENT_FETCHED: [CrawlerEventManager] CREATED_ROBOTS_META: [CrawlerEventManager] URLS_EXTRACTED: [CrawlerEventManager] DOCUMENT_IMPORTED: [CrawlerEventManager] DOCUMENT_COMMITTED_ADD: [CrawlerEventManager] REJECTED_NOTFOUND: [AbstractCrawler] test-crawler: Re-processing orphan references (if any)...INFO [AbstractCrawler] test-crawler: Reprocessed 0 orphan references...INFO [AbstractCrawler] test-crawler: 2 reference(s) processed.INFO [CrawlerEventManager] CRAWLER_FINISHEDINFO [AbstractCrawler] test-crawler: Crawler completed.INFO [AbstractCrawler] test-crawler: Crawler executed in 6 seconds.INFO [MapDBCrawlDataStore] Closing reference store: ./work/crawlstore/mapdb/test-crawler/INFO [JobSuite] Running test-crawler: END (Fri Jan 08 16:21:17 CET 2016)">INFO [AbstractCollectorConfig] Configuration loaded: id=test-collector; logsDir=./logs; progressDir=./progressINFO [JobSuite] JEF work directory is: ./progressINFO [JobSuite] JEF log manager is : FileLogManagerINFO [JobSuite] JEF job status store is : FileJobStatusStoreINFO [AbstractCollector] Suite of 1 crawler jobs created.INFO [JobSuite] Initialization...INFO [JobSuite] No previous execution detected.INFO [JobSuite] Starting execution.INFO [AbstractCollector] Version: Norconex HTTP Collector 2.4.0-SNAPSHOT (Norconex Inc.)INFO [AbstractCollector] Version: Norconex Collector Core 1.4.0-SNAPSHOT (Norconex Inc.)INFO [AbstractCollector] Version: Norconex Importer 2.5.0-SNAPSHOT (Norconex Inc.)INFO [AbstractCollector] Version: Norconex JEF 4.0.7 (Norconex Inc.)INFO [AbstractCollector] Version: Norconex Committer Core 2.0.3 (Norconex Inc.)INFO [JobSuite] Running test-crawler: BEGIN (Fri Jan 08 16:21:17 CET 2016)INFO [MapDBCrawlDataStore] Initializing reference store ./work/crawlstore/mapdb/test-crawler/INFO [MapDBCrawlDataStore] ./work/crawlstore/mapdb/test-crawler/: Done initializing databases.INFO [HttpCrawler] test-crawler: RobotsTxt support: trueINFO [HttpCrawler] test-crawler: RobotsMeta support: trueINFO [HttpCrawler] test-crawler: Sitemap support: trueINFO [HttpCrawler] test-crawler: Canonical links support: trueINFO [HttpCrawler] test-crawler: User-Agent: INFO [SitemapStore] test-crawler: Initializing sitemap store...INFO [SitemapStore] test-crawler: Done initializing sitemap store.INFO [HttpCrawler] 1 start URLs identified.INFO [CrawlerEventManager] CRAWLER_STARTEDINFO [AbstractCrawler] test-crawler: Crawling references...INFO [CrawlerEventManager] DOCUMENT_FETCHED: [CrawlerEventManager] CREATED_ROBOTS_META: [CrawlerEventManager] URLS_EXTRACTED: [CrawlerEventManager] DOCUMENT_IMPORTED: [CrawlerEventManager] DOCUMENT_COMMITTED_ADD:

Gadget InspectorThis project inspects Java libraries and classpaths for gadget chains. Gadgets chains are used to construct exploits for deserialization vulnerabilities. By automatically discovering possible gadgets chains in an application's classpath penetration testers can quickly construct exploits and application security engineers can assess the impact of a deserialization vulnerability and prioritize its remediation.This project was presented at Black Hat USA 2018. Learn more about it there! (Links pending)DISCLAIMER: This project is alpha at best. It needs tests and documentation added. Feel free to help by adding either!BuildingAssuming you have a JDK installed on your system, you should be able to just run ./gradlew shadowJar. You can then run the application with java -jar build/libs/gadget-inspector-all.jar .How to UseThis application expects as argument(s) either a path to a war file (in which case the war will be exploded and all of its classes and libraries used as a classpath) or else any number of jars.Note that the analysis can be memory intensive (and so far gadget inspector has not been optimized at all to be less memory greedy). For small libraries you probably want to allocate at least 2GB of heap size (i.e. with the -Xmx2G flag). For larger applications you will want to use as much memory as you can spare.The toolkit will go through several stages of classpath inspection to build up datasets for use in later stages. These datasets are written to files with a .dat extension and can be discarded after your run (they are written mostly so that earlier stages can be skipped during development).After the analysis has run the file gadget-chains.txt will be written.ExampleThe following is an example from running against commons-collections-3.2.1.jar, e.g. withwget -Xmx2G -jar build/libs/gadget-inspector-all.jar commons-collections-3.2.1.jarIn gadget-chains.txt there is the following chain:com/sun/corba/se/spi/orbutil/proxy/CompositeInvocationHandlerImpl.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (-1) com/sun/corba/se/spi/orbutil/proxy/CompositeInvocationHandlerImpl.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (0) org/apache/commons/collections/map/DefaultedMap.get(Ljava/lang/Object;)Ljava/lang/Object; (0) org/apache/commons/collections/functors/InvokerTransformer.transform(Ljava/lang/Object;)Ljava/lang/Object; (0) java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0)The entry point of this chain is an implementation of the JDK InvocationHandler class. Using the same trick as in the original commons-collections gadget chain, any serializable implementation of this class is reachable in a gadget chain, so the discovered chain starts here. This method invokes classToInvocationHandler.get(). The discovered gadget chain indicates that the classToInvocationHandler can be serialized as a DefaultedMap so that the this invocation jumps to DefaultedMap.get(). The next step in the chain invokes value.transform() from this method. The parameter value in this class can be serialized as a InvokerTransformer. Inside this class's transform method we see that we call cls.getMethodName(iMethodName, ...).invoke(...).

GATINEAU, QC, CANADA – Thursday, August 25, 2014 – Norconex is announcing the launch of Norconex Filesystem Collector, providing organizations with a free “universal” filesystem crawler. The Norconex Filesystem Collector enables document indexing into target repositories of choice, such as enterprise search engines.Following on the success of Norconex HTTP Collector web crawler, Norconex Filesystem Collector is the second open source crawler contribution to the Norconex “Collector” suite. Norconex believes this crawler allows customers to adopt a full-featured enterprise-class local or remote file system crawling solution that outlasts their enterprise search solution or other data repository.“This not only facilitates any future migrations but also allows customer addition of their own ETL logic into a very flexible crawling architecture, whether using Autonomy, Solr/LucidWorks, ElasticSearch, or any others data repository,” said Norconex President Pascal Essiembre.Norconex Filesystem Collector AvailabilityNorconex Filesystem Collector is part of Norconex’s commitment to deliver quality open-source products, backed by community or commercial support. Norconex Filesystem Collector is available for immediate download at /collectors/collector-filesystem/download.Founded in 2007, Norconex is a leader in enterprise search and data discovery. The company offers a wide range of products and services designed to help with the processing and analyzing of structured and unstructured data.For more information on Norconex Filesystem Collector:Website: /collectors/collector-filesystemEmail: [email protected]###Pascal Essiembre has been a successful Enterprise Application Developer for several years before founding Norconex in 2007 and remaining its president to this day. Pascal has been responsible for several successful Norconex enterprise search projects across North America. Pascal is also heading the Product Division of Norconex and leading Norconex Open-Source initiatives.