Package org.apache.tika.pipes.fetcher.http

Class HttpFetcher

java.lang.Object
org.apache.tika.pipes.fetcher.AbstractFetcher
org.apache.tika.pipes.fetcher.http.HttpFetcher
All Implemented Interfaces:
Initializable, Fetcher, RangeFetcher
public class HttpFetcher extends AbstractFetcher implements Initializable, RangeFetcher
Based on Apache httpclient

  • Field Details

    • HTTP_HEADER_PREFIX

      public static String HTTP_HEADER_PREFIX

    • HTTP_FETCH_PREFIX

      public static String HTTP_FETCH_PREFIX

    • HTTP_STATUS_CODE

      public static Property HTTP_STATUS_CODE
      http status code

    • HTTP_NUM_REDIRECTS

      public static Property HTTP_NUM_REDIRECTS
      Number of redirects

    • HTTP_TARGET_URL

      public static Property HTTP_TARGET_URL
      If there were redirects, this captures the final URL visited

    • HTTP_TARGET_IP_ADDRESS

      public static Property HTTP_TARGET_IP_ADDRESS

    • HTTP_FETCH_TRUNCATED

      public static Property HTTP_FETCH_TRUNCATED

    • HTTP_CONTENT_ENCODING

      public static Property HTTP_CONTENT_ENCODING

    • HTTP_CONTENT_TYPE

      public static Property HTTP_CONTENT_TYPE

  • Constructor Details

    • HttpFetcher

      public HttpFetcher()

    • HttpFetcher

      public HttpFetcher(HttpFetcherConfig httpFetcherConfig)

  • Method Details

    • fetch

      public InputStream fetch(String fetchKey, Metadata metadata, ParseContext parseContext) throws IOException, TikaException
      Specified by:
      fetch in interface Fetcher
      Throws:
      IOException
      TikaException

    • fetch

      public InputStream fetch(String fetchKey, long startRange, long endRange, Metadata metadata, ParseContext parseContext) throws IOException, TikaException
      Specified by:
      fetch in interface RangeFetcher
      Throws:
      IOException
      TikaException

    • setUserName

      @Field public void setUserName(String userName)

    • setPassword

      @Field public void setPassword(String password)

    • setNtDomain

      @Field public void setNtDomain(String domain)

    • setAuthScheme

      @Field public void setAuthScheme(String authScheme)

    • setProxyHost

      @Field public void setProxyHost(String proxyHost)

    • setProxyPort

      @Field public void setProxyPort(int proxyPort)

    • setConnectTimeout

      @Field public void setConnectTimeout(int connectTimeout)

    • setRequestTimeout

      @Field public void setRequestTimeout(int requestTimeout)

    • setSocketTimeout

      @Field public void setSocketTimeout(int socketTimeout)

    • setMaxConnections

      @Field public void setMaxConnections(int maxConnections)

    • setMaxConnectionsPerRoute

      @Field public void setMaxConnectionsPerRoute(int maxConnectionsPerRoute)

    • setMaxSpoolSize

      @Field public void setMaxSpoolSize(long maxSpoolSize)
      Set the maximum number of bytes to spool to a temp file. If this value is -1, the full stream will be spooled to a temp file

      Default size is -1.

      Parameters:
      maxSpoolSize -

    • setMaxRedirects

      @Field public void setMaxRedirects(int maxRedirects)

    • setHttpRequestHeaders

      @Field public void setHttpRequestHeaders(List<String> headers)
      Which http request headers should we send in the http fetch requests.
      Parameters:
      headers - The headers to add to the HTTP GET requests.

    • parseHeaders

      public static Map<String,Collection<String>> parseHeaders(String headersString)

    • setHttpHeaders

      @Field public void setHttpHeaders(List<String> headers)
      Which http headers should we capture in the metadata. Keys will be prepended with HTTP_HEADER_PREFIX
      Parameters:
      headers -

    • setOverallTimeout

      @Field public void setOverallTimeout(long overallTimeout)
      This sets an overall timeout on the request. If a server is super slow or the file is very long, the other timeouts might not be triggered.
      Parameters:
      overallTimeout -

    • setMaxErrMsgSize

      @Field public void setMaxErrMsgSize(int maxErrMsgSize)

    • setUserAgent

      @Field public void setUserAgent(String userAgent)
      When making the request, what User-Agent is sent in the request. By default httpclient adds e.g. "Apache-HttpClient/4.5.13 (Java/x.y.z)"
      Parameters:
      userAgent -

    • setJwtIssuer

      @Field public void setJwtIssuer(String jwtIssuer)

    • setJwtSubject

      @Field public void setJwtSubject(String jwtSubject)

    • setJwtExpiresInSeconds

      @Field public void setJwtExpiresInSeconds(int jwtExpiresInSeconds)

    • setJwtSecret

      @Field public void setJwtSecret(String jwtSecret)

    • setJwtPrivateKeyBase64

      @Field public void setJwtPrivateKeyBase64(String jwtPrivateKeyBase64)

    • initialize

      public void initialize(Map<String,Param> params) throws TikaConfigException
      Specified by:
      initialize in interface Initializable
      Parameters:
      params - params to use for initialization
      Throws:
      TikaConfigException

    • checkInitialization

      public void checkInitialization(InitializableProblemHandler problemHandler) throws TikaConfigException
      Specified by:
      checkInitialization in interface Initializable
      Parameters:
      problemHandler - if there is a problem and no custom initializableProblemHandler has been configured via Initializable parameters, this is called to respond.
      Throws:
      TikaConfigException

    • setHttpClientFactory

      public void setHttpClientFactory(HttpClientFactory httpClientFactory)

    • setHttpClient

      public void setHttpClient(org.apache.http.client.HttpClient httpClient)

    • getHttpClient

      public org.apache.http.client.HttpClient getHttpClient()

    • getHttpFetcherConfig

      public HttpFetcherConfig getHttpFetcherConfig()

    • setHttpFetcherConfig

      public void setHttpFetcherConfig(HttpFetcherConfig httpFetcherConfig)