List of questions
Related questions
Question 73 - DEA-C01 discussion
As Data Engineer, you have requirement to Load set of New Product Files containing Product relevant information into the Snowflake internal tables, Later you analyzed that some of the Source files are already loaded in one of the historical batch & for that you have prechecked Metadata col-umn LAST_MODIFIED date for a staged data file & found out that LAST_MODIFIED date is older than 64 days for few files and the initial set of data was loaded into the table more than 64 days earlier, Which one is the best approach to Load Source data files with expired load metadata along with set of files whose metadata might be available to avoid data duplication?
A.
Since the initial set of data for the table (i.e. the first batch after the table was created) was loaded, we can simply use the COPY INTO command to load all the product files with the known load status irrespective of their column LAST_MODIFIED date values.
B.
The COPY command cannot definitively determine whether a file has been loaded al-ready if theLAST_MODIFIED date is older than 64 days and the initial set of data was loaded into the table more than 64 days earlier (and if the file was loaded into the table, that also occurred more than 64 days earlier). In this case, to prevent accidental reload, the command skips the product files by default.
C.
Set the FORCE option to load all files, ignoring load metadata if it exists.
D.
To load files whose metadata has expired, set the LOAD_UNCERTAIN_FILES copy option to true.
Your answer:
0 comments
Sorted by
Leave a comment first