Package org.apache.tika.metadata.writelimiter

Class StandardMetadataLimiter

java.lang.Object
org.apache.tika.metadata.writelimiter.StandardMetadataLimiter
All Implemented Interfaces:
Serializable, MetadataWriteLimiter
public class StandardMetadataLimiter extends Object implements MetadataWriteLimiter, Serializable
Standard implementation of MetadataWriteLimiter that limits the amount of metadata a parser can add based on maxTotalEstimatedSize, maxFieldSize, maxValuesPerField, and maxKeySize. This can also be used to limit which fields are stored in the metadata object at write-time with includeFields.

All sizes are measured in UTF-16 bytes. The size is estimated as a rough order of magnitude of what is required to store the string in memory in Java. We recognize that Java uses more bytes to store length, offset etc. for strings, but the extra overhead varies by Java version and implementation, and we just need a basic estimate. We also recognize actual memory usage is affected by interning strings, etc. Please forgive us ... or consider writing your own limiter. :)

NOTE: Fields in ALWAYS_SET_FIELDS are always set no matter the current state of maxTotalEstimatedSize. Except for TikaCoreProperties.TIKA_CONTENT, they are truncated at maxFieldSize, and their sizes contribute to the maxTotalEstimatedSize.

NOTE: Fields in ALWAYS_ADD_FIELDS are always added no matter the current state of maxTotalEstimatedSize. Except for TikaCoreProperties.TIKA_CONTENT, each addition is truncated at maxFieldSize, and their sizes contribute to the maxTotalEstimatedSize.

This class uses minimumMaxFieldSizeInAlwaysFields to protect the ALWAYS_ADD_FIELDS and ALWAYS_SET_FIELDS. If we didn't have this and a user sets the maxFieldSize to, say, 10 bytes, the internal parser behavior would be broken because parsers rely on HttpHeaders.CONTENT_TYPE to determine which parser to call.

NOTE: as with Metadata, this object is not thread safe.

Since:
Apache Tika 4.0 (renamed from StandardWriteFilter)
See Also:

  • Field Details

    • ALWAYS_SET_FIELDS

      public static final Set<String> ALWAYS_SET_FIELDS

    • ALWAYS_ADD_FIELDS

      public static final Set<String> ALWAYS_ADD_FIELDS

  • Constructor Details

    • StandardMetadataLimiter

      protected StandardMetadataLimiter(int maxKeySize, int maxFieldSize, int maxEstimatedSize, int maxValuesPerField, Set<String> includeFields, Set<String> excludeFields, boolean includeEmpty)
      Parameters:
      maxKeySize - maximum key size in UTF-16 bytes-- keys will be truncated to this length; if less than 0, keys will not be truncated
      maxEstimatedSize - maximum total estimated size in UTF-16 bytes
      includeFields - if null or empty, all fields are included; otherwise, which fields to add to the metadata object.
      excludeFields - these fields will not be included (unless they're in ALWAYS_SET_FIELDS)
      includeEmpty - if true, this will set or add an empty value to the metadata object.

  • Method Details

    • set

      public void set(String field, String value, Map<String,String[]> data)
      Description copied from interface: MetadataWriteLimiter
      Based on the field and the value, this limiter modifies the field and/or the value to something that should be set in the Metadata object.
      Specified by:
      set in interface MetadataWriteLimiter
      Parameters:
      field - the metadata field name
      value - the value to set
      data - the metadata map to modify

    • add

      public void add(String field, String value, Map<String,String[]> data)
      Description copied from interface: MetadataWriteLimiter
      Based on the field and value, this limiter modifies the field and/or the value to something that should be added to the Metadata object. If the value is null, no value is set or added. Status updates (e.g. write limit reached) can be added directly to the underlying metadata.
      Specified by:
      add in interface MetadataWriteLimiter
      Parameters:
      field - the metadata field name
      value - the value to add
      data - the metadata map to modify