com.esri.gpt.framework.http.crawl
Class HttpCrawlRequest

java.lang.Object
  com.esri.gpt.framework.http.HttpClientRequest
      com.esri.gpt.framework.http.crawl.HttpCrawlRequest

public class HttpCrawlRequest
extends HttpClientRequest

Http crawl request.

Nested Class Summary

Nested classes/interfaces inherited from class com.esri.gpt.framework.http.HttpClientRequest

HttpClientRequest.MethodName

Field Summary

Fields inherited from class com.esri.gpt.framework.http.HttpClientRequest

DEFAULT_CONNECTION_TIMEOUT, DEFAULT_RESPONSE_TIMEOUT

Constructor Summary

HttpCrawlRequest(Bots robotsTxt)
Creates instance of the request.

Method Summary

void	execute() Executes the HTTP request.
String	getUrl() Gets the URL for the request.
protected Long	resolveThrottleDelay(Bots bots) Gets throttle delay.

Methods inherited from class com.esri.gpt.framework.http.HttpClientRequest

createMethod, getBatchHttpClient, getConnectionTimeOutMs, getContentHandler, getContentProvider, getCredentialProvider, getExecutionLog, getMethodName, getResponseInfo, getResponseTimeOutMs, getRetries, newRequest, newRequest, readResponseAsCharacters, setBatchHttpClient, setConnectionTimeMs, setContentHandler, setContentProvider, setCredentialProvider, setMethodName, setRequestHeader, setResponseInfo, setResponseTimeOutMs, setRetries, setUrl

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HttpCrawlRequest

public HttpCrawlRequest(Bots robotsTxt)

Creates instance of the request.

Parameters:

robotsTxt - robots.txt or null if robots not available

Method Detail

execute

public void execute()
             throws IOException

Description copied from class: HttpClientRequest

Executes the HTTP request.

Overrides:

execute in class HttpClientRequest

Throws:

IOException - if an Exception occurs

getUrl

public String getUrl()

Description copied from class: HttpClientRequest

Gets the URL for the request.

Overrides:

getUrl in class HttpClientRequest

Returns:

the request URL

resolveThrottleDelay

protected Long resolveThrottleDelay(Bots bots)

Gets throttle delay.

Throttle delay might be a result of "Crawl-Delay" value read from robots.txt or it might be override to implement custom throttle policy.

Parameters:

bots - robots.txt or null if robots.txt unavailable

Returns:

throttle delay in milliseconds or null if no throttling required.