![]() Note currently Copy activity doesn't support LZO when read/write Parquet files. Supported types are " none", " gzip", " snappy" (default), and " lzo". When reading from Parquet files, Data Factories automatically determine the compression codec based on the file metadata. ![]() The compression codec to use when writing to Parquet files. See details in connector article -> Dataset properties section. Each file-based connector has its own location type and supported properties under location. The type property of the dataset must be set to Parquet. This section provides a list of properties supported by the Parquet dataset. Dataset propertiesįor a full list of sections and properties available for defining datasets, see the Datasets article. ![]() By default, the service uses min 64 MB and max 1G. This means that JVM will be started with Xms amount of memory and will be able to use a maximum of Xmx amount of memory. The flag Xms specifies the initial memory allocation pool for a Java Virtual Machine (JVM), while Xmx specifies the maximum memory allocation pool. If you copy data to/from Parquet format using Self-hosted Integration Runtime and hit error saying "An error occurred when invoking java, message: :Java heap space", you can add an environment variable _JAVA_OPTIONS in the machine that hosts the Self-hosted IR to adjust the min/max heap size for JVM to empower such copy, then rerun the pipeline.Įxample: set variable _JAVA_OPTIONS with value -Xms256m -Xmx16g. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |